• Topology-aware dynamic high-order graph learning proves robustness & high performance. • Dynamic high-order graph module enables boundary & spatial topology perception. • Spectral-spatial image convolution module reduces redundant band & mine local feature. • Spectral-spatial feature fusion module captures rich spectral-spatial features. Hyperspectral image (HSI) classification is challenging due to intra-class spectral variability caused by mixed pixels at land cover boundaries in complex spatial distributions. Additionally, the classification model easily suffers from overfitting due to high-dimensional features and limited available samples. Recently, GCNs have emerged to handle these challenges, as they enable convolution on non-Euclidean data with arbitrary structures and modeling topological relationships between nodes. However, existing GCNs either use predefined adjacency matrices that cannot be updated throughout the training process or perform feature aggregation only between the central node and its first-order neighbors, which hinders the building of spatial topology. This paper introduces a novel topology-aware dynamic high-order graph learning (TDHGL) to address the aforementioned issues. Specifically, we first introduce a spectral-spatial image convolution module (SICM) to reduce spectral redundancy and extract local spectral-spatial features. Then, we design a dynamic high-order graph module (DHGM) to extract the spatial topology of the HSI. It allows for dynamic updating of adjacency matrices and facilitates high-order message interactions between the central node and its higher-order neighbors, mitigating classification inaccuracies induced by intra-class spectral variability while enhancing the TDHGL’s generalization capacity. Finally, the spectral-spatial feature fusion module (SFFM) aggregates the local spectral-spatial features with spatial topology to capture rich spectral-spatial features. Our TDHGL achieves overall accuracies of 94.37 %, 97.17 %, 95.87 %, and 98.21 % on the Indian Pines, Salinas, University of Pavia, and WHU-Hi-LongKou datasets, respectively, and outperforms other state-of-the-art methods.

Invariant feature extraction is a critical challenge in intelligent image processing, particularly with the rapid advancement of multi-source/modal imaging. Cross-modal matching has attracted considerable attention, yet current studies primarily focus on targeted modalities rather than realizing a general approach. In this paper, cross-arbitrary-modal image invariant feature extraction and matching is studied. Inspired by human vision, a purely handcrafted invariant feature transform is proposed for universal cross-modal image matching, named Hierarchical Intrinsic Major Orientation (HIMO). Based on orientation information, a full-chain non-data-driven algorithm is designed that hinges on an Intrinsic Major Orientation (IMO) extraction. The HIMO incorporates a novel keypoint detector utilizing Difference-of-Feature Suppression (DoFS), a Polar-Pyramid descriptor (PolarP), and a Cascaded Dynamic Multi-scale Strategy (CDMS) to effectively address common challenges such as intensity distortion, rotation, scale differences, geometric deformation, and image noise. To validate the proposed method, two massive cross-modal datasets-General Cross-modal Zone (GCZ) and Wide-area Diverse Sources (WDS)-are introduced, alongside two practical evaluation metrics. Comprehensive experiments compared with 10 traditional and 15 deep-learning state-of-the-art algorithms on 5 datasets fully demonstrate that the proposed HIMO achieves superior performance in terms of robustness, stability, and generalization across diverse imaging conditions.

The problem of robust matrix completion-the recovery of a low-rank matrix and a sparse matrix from a sampling of their superposition-has been addressed extensively in prior literature. Yet, much of this work has focused exclusively on the case in which the matrix sampling is done at random, as this scenario is amenable to theoretical analysis. In contrast, sampling with an arbitrary deterministic pattern is often more accommodating to hardware implementation; consequently, the problem of robust matrix completion under deterministic sampling is considered. To this end, a restricted approximate isometry property is proposed and used, along with a modified golfing scheme and a slightly strengthened incoherence condition, to prove that the latent low-rank and sparse matrices are uniquely recoverable via convex optimization with asymptotically high probability, providing the first exact-recovery theory for robust matrix completion with arbitrary deterministic sampling. A corresponding convex-optimization algorithm, driven by a traditional nuclear norm, is developed and then subsequently generalized by substituting a convolutional nuclear norm in order to cover a broader range of application scenarios. Empirical experiments on synthetic data verify the proposed theory while a battery of results on real-world images demonstrate the practical efficacy of the generalized algorithm for robust matrix recovery.

Hyperspectral camouflaged object tracking remains a significant challenge due to the high similarity between objects and replicas in texture and color. Despite recent progress, the bias present in the tracker and the embedding token hinders the model training. Specifically, most methods rely on false-color three-channel images to fine-tune RGB-based trackers. However, it introduces a confounding effect within the RGB domain, potentially leading to harmful biases that misguide the model toward spurious correlations while neglecting the critical spectral discrimination inherent in hyperspectral images. Furthermore, current token-type embedding methods overlook the key correlations between templates and searches, ultimately confusing correlation and impairing tracking performance. To address these challenges, this paper proposes a new unbiased tracking framework named Causal HyperPrompter. It first introduces a structural causal model to disentangle and control exclusive causal factors during tracking, and incorporates a counterfactual intervention strategy to eliminate confounding variables and mitigate the bias inherited from RGB-based models. In addition, we present a novel token-type embedding module that integrates local spectral angle modeling to enhance the semantic link between template and search tokens, thereby improving the model's sensitivity to object localization. Lastly, to overcome the difficulty of manually initializing the bounding box and addressing data scarcity, we introduce a large-scale hyperspectral camouflaged object detection and tracking dataset, BihoT-130 k, consisting of 1,30,750 annotated frames across various camouflage scenes. Extensive experiments on multiple large-scale datasets illustrate the effectiveness of our proposed methods.

Frequency domain analysis reveals fundamental image patterns difficult to observe in raw pixel values, while avoiding redundant information in original image processing. Although recent remote sensing foundation models (FMs) have made progress in leveraging spatial and spectral information, they have limitations in fully utilizing frequency characteristics that capture hidden features. Existing FMs that incorporate frequency properties often struggle to maintain connections with the original image content, creating a semantic gap that affects downstream performance. To address these challenges, we propose the All-in-One Spectral-Spatial-Frequency Awareness Foundation Model (Alliance), a framework that effectively integrates information across all three domains. Alliance introduces several key innovations: (1) a progressive frequency decoding mechanism inspired by human visual cognition that minimizes multi-domain information gaps while preserving connections between general image information and frequency characteristics, progressively reconstructing from low to mid to high frequencies to extract patterns difficult to observe in raw pixel values; (2) a triple-domain fusion attention module that separately processes amplitude, phase, and spectral-spatial relationships for comprehensive feature integration; and (3) frequency embedding with frequency-aware Cls token initialization and frequency-specific mask token initialization that achieves fine-grained modeling of different frequency band information. Additionally, to evaluate FMs generalizability, we construct the Yellow River dataset, a large-scale multi-temporal collection that introduces challenging cross-domain tasks and establishes more rigorous standards for FMs assessment. Extensive experiments across six downstream tasks demonstrate Alliance's superior performance.

Deep learning-based methods have achieved significant success in remote sensing Earth observation data analysis. Numerous feature fusion techniques address multimodal remote sensing image classification by integrating global and local features. However, these techniques often struggle to extract structural and detail features from heterogeneous and redundant multimodal images, particularly in label-scarce scenarios. With the goal of introducing frequency domain learning to model key and sparse detail features, this paper introduces the spatial–spectral-frequency interaction network (S 2 Fin), which integrates pairwise fusion modules across the spatial, spectral, and frequency domains. Specifically, we propose a high-frequency sparse enhancement transformer to refine spectral signatures by adaptively enhancing discriminative high-frequency components. For spatial-frequency interaction, we present a depth-wise strategy: the adaptive frequency channel module fuses low-frequency structural information with enhanced details in shallow layers, while the high-frequency resonance mask amplifies modality-consistent regions in deep layers using phase similarity. In addition, a spatial–spectral attention fusion module bridges the gap between spectral and spatial branches at intermediate depths. Extensive experiments on four benchmark datasets demonstrate that S 2 Fin exhibits good robustness and generalization, and its performance significantly outperforms state-of-the-art methods in few-sample settings. The code is available at https://github.com/HaoLiu-XDU/SSFin .

Multispectral remote sensing image (MSI) semantic segmentation faces challenges of limited labeled data and significant scene variability. Although domain adaptation (DA) and domain generalization (DG) methods alleviate these issues to some extent, they still have limitations. DA requires target domain (TD) data, and DG has limited task adaptability. The recently emerged segment anything model (SAM) demonstrates exceptional zero-shot generalization capabilities, yet its visible-light training data and interactive prompt requirements prevent direct application to MSI segmentation tasks. To address these challenges, this article proposes a single-source domain defect-aware adaptation and style-modulated generalization network (SDSnet), which integrates two key innovations: defect-aware prompt learning that automatically focuses on high-difficulty regions through entropy-based defect detection, and style generalization learning that enhances cross-domain adaptability via codebook-based style modulation. Through knowledge distillation, SDSnet enables efficient inference using only the base network, without additional computational overhead. Extensive experiments on three TDs demonstrate SDSnet's superiority over state-of-the-art DA, DG, and SAM-based methods. Code will be available at https://github.com/zhaoboyu34526/SDSnet.

Multi-modal image segmentation has recently attracted considerable attention due to its ability to integrate complementary information from diverse sensors, thereby enabling more accurate semantic predictions in complex or specialized scenarios. However, as data volume and model capacity continue to grow, many existing methods suffer substantial increases in parameters and computational costs, particularly with the widespread adoption of Vision Foundation Models (VFMs). To address these challenges, we introduce a Lightweight Multi-modal Fine-Tuning framework (LiteMFT) designed for efficient and generalizable adaptation of RGB-pretrained VFMs to multi-modal semantic segmentation. By incorporating only a small number of trainable parameters, LiteMFT enables effective extension of existing models to handle multi-modal image fusion tasks. The framework centers around two key components: the Modality Local Competition (MLC) module, which dynamically and efficiently fuses complementary features across modalities, and the Gated Low-Rank Adapter (GLR), which improves the backbone's adaptability to multi-modal data through content-aware low-rank transformation. Extensive experiments on both bi-modal and tri-modal segmentation tasks demonstrate that LiteMFT not only achieves competitive or superior performance but also exhibits strong scalability for additional modalities, underscoring its practicality and broad applicability in multi-modal semantic segmentation.

Feature reconstruction networks have achieved remarkable performance in few-shot fine-grained classification tasks. Nonetheless, traditional feature reconstruction networks rely on linear regression. This linearity may cause the loss of subtle discriminative cues, ultimately resulting in less precise reconstructed features. Moreover, in situations where the background predominantly occupies the image, the background reconstruction errors tend to overshadow foreground reconstruction errors, resulting in inaccurate reconstruction errors. In order to address the two key issues, a novel approach called the Foreground-Aware Kernelized Feature Reconstruction Network (FKFRN) is proposed. Specifically, to address the problem of imprecise reconstructed features, we introduce kernel methods into linear feature reconstruction, extending it to nonlinear feature reconstruction, thus enabling the reconstruction of richer, finer-grained discriminative features. To tackle the issue of inaccurate reconstruction errors, the foreground-aware reconstruction error is proposed. Specifically, the model assigns higher weights to features containing more foreground information and lower weights to those dominated by background content, which reduces the impact of background errors on the overall reconstruction. To estimate these weights accurately, we design two complementary strategies: an explicit probabilistic graphical model and an implicit neural network-based approach. Extensive experimental results on eight datasets validate the effectiveness of the proposed approach for few-shot fine-grained classification.

The Global Positioning System (GPS) provides precise geographic locations for our vehicles. Nevertheless, it is frequently subject to disruptions, potentially resulting in incomplete or absent trajectory data. To address this challenge, we present DictRoadNet, a framework designed for GPS trajectory completion, which uses a clustering-based dictionary module for initial trajectory generation and a road network module for refining results based on road network data. First, we introduce a dictionary that employs a clustering-based strategy for selecting key-value pairs, which can be used in GPS data processing. This dictionary can provide auxiliary general information acquired from trajectory clusters, enhancing the generation of rational trajectories with additional details. Second, we propose a Road Network Module that utilizes a directed graph to store road network information derived from historical GPS trajectories. This module refines the output by aligning it with an empirically constructed road network, ensuring that trajectory completions are plausible and closely adhere to actual road paths. We achieved enhancements across all tasks when assessed against Average and Final Displacement Error, with the highest enhancement reaching up to 9.50% compared to state-of-the-art methods.

Hyperspectral imaging delivers high-resolution spectral-spatial information to support molecular tissue characterization, but its clinical utility is far from being fully realized. Existing segmentation techniques are constrained by fixed or suboptimal band selection strategies and insufficient frequency-domain modeling, which limit their ability to fully exploit discriminative spectral cues and subtle tissue structures. To address these challenges, we propose AMBS-SF2Net, a unified framework that enhances spectral representation and hierarchical frequency modeling for accurate and efficient segmentation. Specifically, the Adaptive Mask-based Band Selection (AMBS) module dynamically identifies informative spectral channels, the Adaptive Spectral-Frequency Integration (ASFI) module fuses multi-scale spatial edges and frequency-aware spectral features, and the Multi-Axis Frequency Enhanced (MAFE) module captures complementary spectral and spatial frequency patterns along different tensor dimensions. To rigorously evaluate the method’s generalizability, we conduct extensive experiments on datasets spanning distinct imaging scales, comprising a microscopic cholangiocarcinoma pathology dataset and two macroscopic tissue datasets of pig abdominal organs and human placenta. Results demonstrate that AMBS-SF2Net significantly outperforms state-of-the-art methods, exhibiting superior robustness across varying spectral resolutions and spatial modalities, thereby validating its strong potential for diverse clinical applications.

To overcome the challenges posed by domain shift in hyperspectral image (HSI) classification, methods based on domain adaptation (DA) have been widely used. Currently, most HSI DA methods focus on designing complex strategies to align the distributions of the source domain (SD) and the target domain (TD) in the feature space after feature extraction, yielding promising results. However, when there exists a large domain shift between SD and TD, it becomes challenging to map them into the same feature space. In this article, we propose the bidirectional mamba and domain mixing network (BMDMnet). Since pure CNN architectures are constrained in local feature extraction, while transformer-based models improve global feature capturing capability at the cost of high computational complexity, we propose the bidirectional mamba module (BMM) as an efficient solution for capturing long-range dependencies. In addition, a self-distillation strategy is employed during training. By utilizing a more stable teacher model, reliable predictions can be obtained in the TD. Subsequently, a domain mixing supervised learning (DMSL) module is designed, which creates a mixed domain by selecting low-entropy sample-pseudo-label pairs from the TD and randomly combining them with sample-label pairs from the SD. DMSL aims to introduce mixed domain to mitigate the inter-domain gap in the data space, thereby enabling the model to learn TD representations more effectively. Experiments demonstrate that BMDMnet outperforms state-of-the-art algorithms across three cross-scene datasets.

Few-shot learning (FSL) has gained increasing attention in hyperspectral image (HSI) classification due to its ability to perform cross-domain classification with minimal labeled samples. However, existing FSL methods overlook the continuity of HSI spectral sequences and fail to utilize the large amount of unlabeled samples in the target domain. To address these issues, we introduce a novel cross-domain HSI classification method that combines self-supervised learning with FSL (SSFSL). This approach uses self-supervised learning and FSL to extract transferable knowledge from the source domain and introduces an adaptive soft label generation algorithm to leverage unlabeled samples in the target domain. Compared to existing cross-domain FSL classification methods, the proposed approach considers the spectral sequence continuity of HSI and effectively extracts useful information from unlabeled samples in the target domain. Extensive experiments conducted on three datasets demonstrate that SSFSL outperforms state-of-the-art methods in both quantitative and qualitative aspects.

Placement is a crucial and time-intensive step in the modern Electronic Design Automation (EDA) chip design process, involving the allocation of millions of modules on a chip canvas. Previous studies have demonstrated the efficacy of machine learning methods, especially reinforcement learning (RL) in chip placement. However, existing RL-based methods still face challenges, including lengthy placement times due to placing only a single module at each step, and a lack of generalization capability. In response to these challenges, we propose a Graph Convolutional Network (GCN)-based diffusion model called GraphDiffusion. Our model presents a novel approach for chip placement by representing the positioning of movable nodes as a conditional denoising diffusion process. The GCN is utilized to extract feature information from the circuit, which then serves as an embedding to guide the diffusion model in generating the chip layout. We exploit the diffusion model’s capabilities to improve layout quality and generalizability. We conduct extensive experiments on the ISPD benchmark. The results demonstrate that our approach outperforms advanced placement methods while enhancing the transferability of the layout model. Upon training with expert data, GraphDiffusion can interpret the input circuit netlists and generate state-of-the-art chip layouts.

High-resolution (HR) remote sensing image change detection aims to identify surface changes; however, complex scenes and irregular object edges pose significant challenges to achieving accurate results. Existing methods leverage upsampling, downsampling, or dilated convolution to capture multiscale spatial features and fuse fine-scale details into coarse-scale features using concatenation, addition, or skip connections to enhance edge information. However, these direct fusion operations can cause fine edge details to be overshadowed by dominant regional features. To address this, we propose an edge-guided change detection (EGCD) network that improves edge preservation and detection accuracy. In the encoding stage, a region-edge feature extraction module (REM) is introduced to extract regional and edge features in parallel using a two-branch structure for each temporal image. The edge and regional features from the two temporal images are then fused independently via a separation feature fusion (SFF) module, preventing fine edge details from being dominated by regional features. In the decoding stage, a edge enhancement upsampling (EEU) module uses edge features to guide the reconstruction of regional features, ensuring precise boundary delineation. Experiments on public datasets validate the effectiveness and robustness of the proposed network.

Point cloud semantic segmentation can enhance the understanding of the production environment and is a crucial component of vision tasks. The efficacy and generalization prowess of deep learning-based segmentation models are inherently contingent upon the quality and nature of the data employed in their training. However, it is often challenging to obtain data with inter-class balance, and training an intelligent segmentation network with the imbalanced data may cause cognitive bias. In this paper, a network framework InvSpaceNet is proposed, which generates an inverse feature space to alleviate the cognitive bias caused by imbalanced data. Specifically, we design a dual-branch training architecture that combines the superior feature representations derived from instance-balanced sampling data with the cognitive corrections introduced by the proposed inverse sampling data. In the inverse feature space of the point cloud generated by the auxiliary branch, the central points aggregated by class are constrained by the contrastive loss. To refine the class cognition in the inverse feature space, features are used to generate point cloud class prototypes through momentum update. These class prototypes from the inverse space are utilized to generate feature maps and structure maps that are aligned with the positive feature space of the main branch segmentation network. The training of the main branch is dynamically guided through gradients back propagated from different losses. Extensive experiments conducted on four large benchmarks (i.e., S3DIS, ScanNet v2, Toronto-3D, and SemanticKITTI) demonstrate that the proposed method can effectively mitigate point cloud imbalance issues and improve segmentation performance.

Hyperspectral super-resolution is commonly accomplished by the fusing of a hyperspectral imaging of low spatial resolution with a multispectral image of high spatial resolution, and many tensor-based approaches to this task have been recently proposed. Yet, it is assumed in such tensor-based methods that the spatial-blurring operation that creates the observed hyperspectral image from the desired super-resolved image is separable into independent horizontal and vertical blurring. Recent work has argued that such separable spatial degradation is ill-equipped to model the operation of real sensors which may exhibit, for example, anisotropic blurring. To accommodate this fact, a generalized tensor formulation based on a Kronecker decomposition is proposed to handle any general spatial-degradation matrix, including those that are not separable as previously assumed. Analysis of the generalized formulation reveals conditions under which exact recovery of the desired super-resolved image is guaranteed, and a practical algorithm for such recovery, driven by a blockwise-group-sparsity regularization, is proposed. Extensive experimental results demonstrate that the proposed generalized tensor approach outperforms not only traditional matrix-based techniques but also state-of-the-art tensor-based methods; the gains with respect to the latter are especially significant in cases of anisotropic spatial blurring.

Hyperspectral image multi-class change detection (HSI-MCD) based on deep learning (DL) rely significantly on the number of labeled data. Due to the high cost of manually labeling for hyperspectral images (HSIs), obtaining a large amount of labeled samples is difficult. Moreover, for multi-class change detection (MCD) tasks, there is the phenomenon of semantic cross-coupling of changes due to complex change scenarios. To solve the above problems, a cross-domain few-shot learning method based on fractional domain information for HSI-MCD (FrCFSL) is proposed. Firstly, a spectral-spatial-fractional information extraction module is proposed, which can extract spectral-spatial-fractional domain joint feature. Thus, the module can obtain more comprehensive and discriminative representations of land cover categories, alleviating the phenomenon of semantic cross-coupling between classes. Afterward, a cross-domain fewshot learning strategy is introduced, where it learns task-relevant category discrimination meta-knowledge from a pair of richly labeled very high-resolution optical images (VHRIs) dataset and transfers it to the bitemporal HSIs dataset. Thus, the model can achieve better MCD performance with a small number of labeled samples. Finally, to mitigate the domain distribution differences between VHRIs data and HSIs data, a topological structure alignment module is proposed to align the intrinsic topological relationships between land cover categories, thus narrowing the gap between the two domain distributions. Through experiments conducted on three HSI-MCD datasets and comparative analysis with six state-of-the-art methods, the validity and stability of the proposed method are indicated.

Utilizing hyperspectral remote sensing technology enables the extraction of fine-grained land cover classes. Typically, satellite or airborne images used for training and testing are acquired from different regions or times, where the same class has significant spectral shifts in different scenes. In this paper, we propose a Bi-directional Domain Adaptation (BiDA) framework for cross-domain hyperspectral image (HSI) classification, which focuses on extracting both domain-invariant features and domain-specific information in the independent adaptive space, thereby enhancing the adaptability and separability to the target scene. In the proposed BiDA, a triple-branch transformer architecture (the source branch, target branch, and coupled branch) with semantic tokenizer is designed as the backbone. Specifically, the source branch and target branch independently learn the adaptive space of source and target domains, a Coupled Multi-head Cross-attention (CMCA) mechanism is developed in coupled branch for feature interaction and inter-domain correlation mining. Furthermore, a bi-directional distillation loss is designed to guide adaptive space learning using inter-domain correlation. Finally, we propose an Adaptive Reinforcement Strategy (ARS) to encourage the model to focus on specific generalized feature extraction within both source and target scenes in noise condition. Experimental results on cross-temporal/scene airborne and satellite datasets demonstrate that the proposed BiDA performs significantly better than some state-of-the-art domain adaptation approaches. In the cross-temporal tree species classification task, the proposed BiDA is more than 3%∼5% higher than the most advanced method. The codes will be available from the website: https://github.com/YuxiangZhang-BIT/IEEE TCSVT BiDA.

Most existing hyperspectral image (HSI) classification methods primarily focus on capturing subtle spectral variations by leveraging local spectral-spatial cues derived from patch-level representations. However, limited attention has been given to exploring the global spatial contextual correlations among pixels of HSI. In this study, we propose the Superpixel Segmentation and Classification Mutual Enhancement Network (S2CMEN), a novel framework that integrates global spatial correlations with spectral information through the mutual enhancement of superpixel segmentation and classification. Specifically, a global spatial adaptive module (GSAM) is designed to obtain the direct correlation of the global classes in HSI. It consists of an Adaptive Spectral-Superpixel Network (ASSN) and a Graph Convolutional Network (GCN), forming a synergistic architecture that effectively captures global spatial relationships by adaptively deriving superpixel results from HSIs. Notably, GSAM offers a transferable global spatial representation for HSI tasks, enabling integration with other spectral feature extraction models. Furthermore, we develop a Spatial-Spectral Fusion Module (SSFM) to obtain comprehensive spectral features and fuse them with the extracted global spatial features. Finally, under the constraint of a unit loss, the Mutual Enhancement Strategy (MES) can make the superpixel segmentation loss and the classification loss mutually enhance each other for better performance. We conducted extensive experiments on three public datasets. The proposed S2CMEN achieves overall classification accuracies of 97.38%, 92.33%, and 91.38% on Indian Pines, Pavia University, and Houston, respectively, consistently surpassing existing state-of-the-art methods.

Change detection (CD) is a critical task for monitoring the spatiotemporal evolution of the Earth’s surface. Recently, due to the advantages of reduced annotation cost and improved labeling efficiency, weakly supervised change detection (WSCD) has attracted increasing attention. However, existing WSCD methods encounter several critical challenges, including incomplete activation of class activation maps (CAMs), interference from noisy pseudo-labels during training, and instability in change recognition caused by illumination and environmental variations. To address these issues, we propose a prototype-aware learning and dual-view regularization network (PDRNet) for image-level WSCD. Specifically, to address the issue of incomplete activation caused by the tendency of CAM to focus excessively on locally discriminative regions, PDRNet devises a prototype-aware module (PAM), which captures stable category prototypes and refines CAM quality by reactivating hierarchical features. Furthermore, to mitigate the network’s sensitivity to noisy pseudo-labels, a dual-view regularization strategy (DRS) is designed to partition pseudo-labels into clean and noisy regions. Region-specific regularization is subsequently employed to improve the robustness of the model against noisy supervision. Finally, to enhance the capability of identifying changed regions, PDRNet constructs a wavelet-based change enhancement module (WCEM) to decompose bi-temporal features into multiple frequency bands. This facilitates the comprehensive utilization of low-frequency structural semantics and high-frequency texture details. Extensive experiments and analyses conducted on three publicly available CD datasets yield the superiority of PDRNet.

Ship recognition in synthetic aperture radar (SAR) images has extensive applications across various fields. However, the substantial intra-class variability and inter-class similarity present inherent challenges to achieving high-precision recognition. Speckle noise in the background reduces the signal-to-noise ratio, complicating the extraction of discriminative characteristics. Additionally, traditional convolutional neural networks-based methods, which rely solely on image processing frameworks without leveraging additional modality information, struggle to accurately represent the features of SAR targets with bright scatters. To address these issues, we propose a prototype-guided multimodal network (PGMNet), marking a pioneering effort to introduce an image-text multimodal fusion processing paradigm into SAR ship recognition. First, a generation strategy of the prototype and the key area is designed to improve the distinguishability between targets and backgrounds. Besides, a prototype-guided alignment module (PGAM) is implemented to assist the network in characterizing key area information, enhancing intra-class feature consistency. Furthermore, a text feature processing branch is incorporated to precisely describe ship size information and effectively integrate image-text multimodal features, reducing intra-class feature distance while enlarging inter-class feature distance. Extensive experiments on the OpenSARShip and FUSARShip datasets demonstrate that the proposed PGMNet achieves state-of-the-art (SOTA) performance. Notably, the accuracy of PGMNet is at least 11% higher than the current SOTA algorithms on the OpenSARShip-VI dataset.

In recent years, the utilization of hyperspectral image (HSI) and light detection and ranging (LiDAR) for collaborative classification has emerged as a significant research direction in earth observation tasks, with diverse joint classification algorithms showing promising performance using varying network architectures. However, these methodologies infrequently address the challenge of fusion arising from the substantially larger volume of HSI feature information compared to LiDAR features. Moreover, the effective learning of HSI and LiDAR features while mitigating modality conflicts remains an area that necessitates further investigation. As such, a Fractional Fourier Enhanced Fusion Network based on Pareto Optimization (FrFENet) is proposed for HSI and LiDAR Data classification. To address the disparity in information volume between modalities, a weighted fractional Fourier enhanced fusion module (WFrFEF) is introduced, which applies a weighted fractional Fourier transform to HSI features, enhancing their representations and facilitating balanced fusion with LiDAR features. Furthermore, a Pareto-based soft optimization strategy, HLPareto, is designed to balance learning rates across HSI and LiDAR features in a dual-branch network, effectively avoiding optimization conflicts. Additionally, a spatial-spectral integration module (SSIM) and an elevation information enhancement module (EIEM) are developed to improve feature extraction. The SSIM enables effective spatial-spectral fusion by facilitating token-level interactions, while the EIEM enhances elevation feature representation, preserving spatial geometric information in LiDAR data. Extensive experiments and comparative analyses conducted on three widely utilized HSI and LiDAR datasets have shown that the proposed FrFENet exhibits superior classification performance.

Deep learning-based methods have made significant progress in hyperspectral target detection (HTD). Unfortunately, limited target prior information and imbalance class resulting from the low occurrence probability of target leaves deep learning-based methods to confront bottlenecks. To ameliorate the abovementioned issues, a Transformer-based cross-domain few-shot learning (TCFSL) method is proposed for HTD. First, the TCFSL leverages cross-domain few-shot learning (FSL) to establish FSL tasks in both the source domain (SD) and the target domain (TD). This allows the TCFSL to learn transferable knowledge of the SD and distinguishable feature embedding model for the TD, to address the problems of target priori lacking and imbalance class. Second, feature-level and distribution-level domain adaptation (DA) is used to tackle the problem of domain shift in cross-domain FSL. The feature-level DA extracts intradomain information of the SD and TD to learn their common features to alleviate domain shift. The distribution-level DA based on cross-Transformer present interdomain distribution-level information aggregation and captures domain similarities of two data domains. By pursuing similarities between two data domains, the distribution-level DA block prompts specific FSL tasks in each domain, facilitating the target detection task. Finally, cross-domain FSL and DA blocks are trained in a unitary manner, which facilitates real-time information interaction and parameter adjustment between different blocks to achieve the optimal model. Experiments conducted on six HSI datasets indicate that the TCFSL outperforms 12 compared methods.

In recent years, research in the field of hyperspectral image classification (HSIC) has increasingly focused on the open-set problem. Open-set classification demands not only accurately classifying the known categories but also identifying the unknown samples that are not labeled or included within the training data during testing stage. Existing open-set methods often suffer from misclassification between the known and unknown categories due to their inadequate utilization of metric space. Moreover, relying on a single threshold strategy performs poorly for identifying unknown categories in complex open environments. In this paper, a fractional domain information enhanced hyperspherical proto-type learning method (FrHSPL) is proposed for hyperspectral image open-set classification. FrHSPL develops a hyperspherical prototype learning (HSPL) strategy that ensures the features of known categories are uniformly distributed on the hypersphere. Therefore, HSPL can effectively enhance inter-class separability and optimize the exploitation of metric space. Subsequently, to enhance the discrimination capability of spectral features, a frequency-spatial-spectral information aggregation module is devised to deeply integrate fractional domain information with spatial and spectral information. Finally, an open-set recognition module is designed to identify unknown categories by using the prototypes of each known category along with the corresponding prototype radii. Extensive experiments on four common HSI datasets indicate that the proposed FrHSPL exhibits superior performance in comparison with both closed-set and open-set methods.

Extracting spatial-spectral joint features has become a critical approach for improving model classification performance in the field of hyperspectral image classification (HSIC). However, existing methods fail to fully exploit nonlinear spatial-spectral information. Unlike traditional convolutional neural networks (CNNs), graph convolutional neural networks (GCNs) can extract nonlinear spatial information. Nevertheless, both methods lack an accurate measurement of local neighborhood information, leading to blurred classification boundaries for ground objects. Additionally, the high-dimensional nature of hyperspectral data results in poor generalization and redundant information of trained models. To address these three issues, a nonlinear weighted graph convolution network based on manifold geometric regularization (MGR-NWGCN) method is devised for HSIC. Specifically, a nonlinear weighted graph convolution (NWGCN) module is designed, which utilizes a Graph-in-Graph structure based on cosine similarity-based normalized weighted graph convolution to extract nonlinear spatial-spectral information. Then, the manifold curvature regularization (C-MGR) module is implemented to improve the accuracy of similarity measurement and to enhance the generalization ability of the model, which constrains the model to form flatter feature manifold surfaces. Finally, the manifold intrinsic dimensionality regularization (ID-MGR) module is developed with the aim of eliminating redundant information, which embeds noise onto the surface of a low-dimensional manifold. The superior classification performance and robustness of the proposed MGR-NWGCN method are validated through extensive experiments on four datasets, with comparisons conducted against nine methods.

Hyperspectral image classification (HSIC) aims to identify land cover categories by leveraging the spectral and spatial information contained in hyperspectral images (HSI). Currently, many deep learning approaches utilize dual-branch networks to process spectral and spatial data separately, followed by the application of specialized modules to facilitate feature interaction or fusion. However, the design of these modules demands considerable time and effort from researchers and may not adequately capture the inherent relationships between independent spatial and spectral features in a dynamic manner. To address these issues, we propose the dynamic stitchable neural network (DSNet) for HSIC. While the DSNet maintains a dual-branch structure, it operates without traditional feature fusion or interaction. Instead, it employs a stitching network approach to integrate the two branches. Specifically, a spatial-spectral stitching module is presented to incorporates multiple stitching layers at various positions between the two network branches, creating new stitched networks that retain the strengths of both original networks. Additionally, a reinforcement learning-based strategy is designed for dynamically selecting stitching positions tailored to specific datasets, enabling the model to adaptively optimize the integration of spatial and spectral features. Recognizing the effectiveness of vision transformer (ViT) in learning spatial information and the capability of 1D convolutional neural network (1DCNN) in capturing spectral details, the DSNet directly stitches these two networks together. This fusion maximizes the utilization of both foundational networks, yielding a new hybrid network that delivers exceptional performance while also alleviating the burden on researchers to develop new architectures from scratch. Extensive experiments and analyses conducted on three public HSI datasets demonstrate the superiority of the proposed method, validating the effectiveness of our innovative modules. The codes of this work will be available from the website: https://github.com/ZZC/IEEE-TGRS-DSNet.

Accurate and comprehensive tree cover mapping plays a vital role in forest management and ecosystem assessment. However, the lack of spectral information and the low quality of available black and white (B&W) imagery makes it difficult to retroactively predict an historical tree cover map using common methods. Here, a network structure called B&WTreeNet, based on semantic segmentation, is proposed to make full use of the limited labeled training remote sensing data to obtain tree cover information from historical datasets. The proposed B&WTreeNet is capable of cross-temporal semantic segmentation, including various data augmentation methods, to address inconsistent tree cover features caused by variations in image quality. The luminance enhancer (LE) and other modules in B&WTreeNet can extract the characteristics of tree cover effectively, successfully compensating for the limited spectral information in B&W image datasets. The countrywide historical tree cover map in Switzerland generated for the 1980s using a limited training dataset from 2018 to 2019 agrees well with manual interpretation results.

Change detection (CD) plays a crucial role in remote sensing (RS) applications. Although deep learning (DL)-based CD methods have achieved impressive performance, they typically rely on large amounts of well-annotated data, which is often scarce in real-world scenarios. This data scarcity leads to overfitting and limited generalization ability in conventional CD models. To address these challenges, this article proposes a novel few-sample CD framework named multiscale style DG method for change detection (MGCD). The core idea is to enhance the model’s ability to learn domain-invariant features by increasing data diversity across multiple style scales. Specifically, MGCD introduces global style diversity by incorporating out-of-domain natural images, and enriches local structural styles through unsupervised clustering and randomization. Additionally, a semantic consistency supervision (SCS) strategy is designed to guide multitemporal feature learning, enabling the network to better capture changes across diverse styles and scales. Extensive experiments conducted on three benchmark datasets demonstrate the effectiveness and robustness of the proposed MGCD framework in few-sample CD tasks.

To leverage the complementary advantages of SAR’s all-weather and all-day imaging capability and optical imagery’s intuitive visualization, SAR-to-optical image translation (S2OIT) has emerged as a promising solution to mitigate the interpretability challenges posed by SAR’s speckle noise and geometric distortions. However, the scale of high-quality registered SAR-optical data is limited, where incorporating priors is a viable solution. What’s more, the digging out of optical prior is insufficient among the existing methods, leading to inadequate synthesis of optical-like texture in translated optical images. To address these challenges, we propose DOGAN, a DINO-based optical-prior-driven GAN framework that integrates ample optical priors extracted from a pretrained DINO model into the S2OIT process. Specifically, to fully exploit the tremendous optical prior preserved in pretrained DINO and extract multiscale optical prior, a DINO-based Optical-prior Extraction (DOE) module is proposed. Furthermore, to elevate the domain adaptability of optical prior, a lightweight Stacked Optical-Aware (SOA) adapter is proposed to fine-tune DINO for remote sensing data with minimal trainable parameters. To instill the extracted affluent optical prior into the S2OIT pipeline stably, the SAR-optical Multi-scale Domain Alignment (SO-MDA) module is proposed, which employs L1 and Multi-kernel Maximum Mean Discrepancy (MK-MMD) losses to align intermediate optical and S2O features. Extensive experiments on SAR2Opt and SEN1-2 datasets demonstrate that DOGAN achieves state-of-the-art performance in both translation fidelity and structural realism. To the best of our knowledge, this is the first work to leverage DINO-based optical priors for the S2OIT task.

Relic landslide, formed over a long period, possess the potential for reactivation, making them a hazardous geological phenomenon. While reliable relic landslide detection benefits the effective monitoring and prevention of landslide disaster, semantic segmentation using high-resolution remote sensing images for relic landslides faces many challenges, including the object visual blur problem, due to the changes of appearance caused by prolonged natural evolution and human activities, and the small-sized dataset problem, due to difficulty in recognizing and labelling the samples. To address these challenges, a semantic segmentation model, termed mask-recovering and interactivefeature- enhancing (MRIFE), is proposed for more efficient feature extraction and separation. Specifically, to address the visual blur problem, a contrastive learning and mask reconstruction approach is designed under the guidance of remote sensing visual interpretation expert knowledge, which states the height variation at the landslide boundary contributing the most to landslide identification. This approach constructs local patches from the landslide boundary and background to perform supervised contrastive learning and applies mask reconstruction to local patches, guiding the model to focus on the landslide boundary and to extract the most contributive local salient features for reliable recognition. Meanwhile, to address the smallsized dataset problem, a self-distillation learning method is introduced, which uses a momentum encoder to update the teacher network with the average of the student network, suppressing overfitting caused by background interference. By constructing contrastive input pairs, the approach increases the diversity and combinations within the contrastive sample space and improves sample utilization. The proposed MRIFE is evaluated on a real relic landslide dataset, and experimental results show that it greatly improves the performance of relic landslide detection. For the semantic segmentation task, compared to the baseline, the precision increases from 0.4226 to 0.5347, the mean intersection over union (IoU) increases from 0.6405 to 0.6680, the landslide IoU increases from 0.3381 to 0.3934, and the F1-score increases from 0.5054 to 0.5646.

Visual grounding for remote sensing (RSVG) aims to detect objects in remote sensing scenes based on textual descriptions. While existing methods perform well on RSVG datasets, they are limited to single-object predictions, making them unsuitable for multi-object candidate category datasets. Open-set methods can be applied to both RSVG and candidate datasets, but their use in remote sensing remains rare. To bridge this gap, we introduce the open-set approach to RSVG and propose Efficient Grounding DINO, using Grounding DINO as a baseline. Open-set methods rely on two key modules: cross-modality fusion and label assignment. Existing cross-modality fusion methods simultaneously update text and multi-scale visual features, which hampers the model’s ability to generalize under different texts and increases learning complexity. Existing methods predict a single object, allowing direct use as a positive example for loss calculation, while open-set methods for multi-objects require one-to-one matching to assign positive and negative samples. However, background interference in the RSVG datasets causes frequent misassignments, slowing model convergence. We address these issues with two innovations: the multi-scale image-to-text fusion module (MSITFM), which updates text features using self-attention to maintain independence from visual features and employs scale-specific cross-attention for multi-scale visual feature fusion to reduce learning complexity, achieving a 3% parameter and 21.6% GFLOPs reduction. Text confidence matching (TCM) incorporates IoU-based confidence into label assignment to reduce mismatches and enhance model performance. Experiments on DIOR-RSVG, RSVG-HR, and DOTA datasets validate the effectiveness of our approach.

Although unsupervised domain adaptation (UDA) has been successfully applied for cross-scene classification of multisource remote sensing (MSRS) data, there are still some tough issues: 1) The vast majority of them are patch-based, requiring pixel by pixel processing at high complexity and ignoring the roles of unlabeled data between different domains. 2) Traditional masked autoencoder (MAE)-based methods lack effective multiscale analysis and require pre-training, ignoring the roles of low-level representations. As such, a hierarchical masked dual-adversarial DA network (HMDA-DANet) is proposed for cross-domain end-to-end classification of MSRS data. Firstly, a hierarchical asymmetric MAE (HAMAE) without pre-training is designed, containing a frequency dynamic large-scale convolutional (FDLConv) block to enhance important structural information in the frequency domain, and an intramodality enhancement and intermodality interaction (IAEIEI) block to embed some additional information beyond the domain distribution by expanding the cross-modal reconstruction space. Representative multimodal multiscale features can be extracted, while to some extent improving their generalization to the target domain. Then, a multimodal multiscale feature fusion (MMFF) block is built to model the spatial and scale dependencies for feature fusion and reduce the layer by layer transmission of redundancy or interference information. Finally, a dual-discriminator-based DA (DDA) block is designed for class-specific semantic feature and global structural alignments in both spatial and prediction spaces. It will enable HAMAE to model the cross-modal, cross-scale, and cross-domain associations, yielding more representative domain-invariant multimodal fusion features. Extensive experiments on five cross-domain MSRS datasets verify the superiority of the proposed HMDA-DANet over other state-of-the-art methods.

With the rapid growth in satellite imagery acquisition and decreasing revisit intervals, efficient on-orbit processing of hyperspectral data has become critical due to limited onboard computing resources. In this context, lifelong learning (LLL) offers a promising solution to enable continuous learning from new data without storing all previous data or retraining from scratch. However, the plasticity-stability dilemma remains a significant challenge, particularly in hyperspectral image (HSI) classification under class-incremental scenarios. To address this, we propose a novel network architecture that integrates contrastive learning and an angular penalty loss. The contrastive learning module facilitates adaptive knowledge fusion, enabling the model to effectively incorporate new information while preserving prior knowledge. The angular penalty loss allows the classifier to dynamically expand for new classes while maintaining discrimination between old and new categories. Together, these components ensure robust knowledge retention, transfer, and adaptability. Experimental results on three benchmark hyperspectral datasets demonstrate that our method significantly outperforms existing approaches, highlighting its efficacy in addressing LLL challenges in HSI classification. The code is available athttps://github.com/Li-ZK/LLL-AFCA.

Moving ship detection is vital for real-time maritime monitoring. Nevertheless, several challenges arise in this area: (i) Wide-area images often need to be sliced into patches to detect tiny targets, which is inefficient. (ii) The ships are small with almost no texture, leading to difficulties in accurate detection. (iii) The contrast between ships and ocean is relatively low, resulting in weak features. Although moving ships exhibit weak features, they often possess distinct wake trails. Capitalizing on this characteristic, we tailored a dual-head supervision network for moving ship detection. Initially, a dual-head supervision architecture is introduced to guide the model in using wake trails for target localization, thereby addressing the inefficiency caused by slicing. Subsequently, the background association head and target confirmation head are introduced to collaboratively enhance detection accuracy by leveraging inter-head attention mechanism. Finally, to address the issues of weak features, the dynamic feature enhancement module is embedded into backbone to boost the model’s feature extraction capability for moving targets. Experiments on GaoFen-1 dataset demonstrated that our method significantly improved the efficiency and performance of infrared moving ship detection and reached the state-of-the-art performance. Source codes will be available at https://github.com/KTqizhi/KTqizhi.github.io.

Fine-grained ship recognition endeavors to accurately locate ship targets and recognize their respective fine-grained categories. Current ship recognition methods primarily rely on the feature pyramid network (FPN) for extracting multiscale features. However, FPN exhibits a spatial misalignment issue when fusing features from adjacent-scale feature maps, leading to an inability to extract fine-grained features. Consequently, this limitation constrains the fine-grained recognition capabilities of these recognition methods. Moreover, ship targets possess a high level of intraclass diversity and interclass similarity, yet existing recognition models struggle to extract features with strong category separability, resulting in weakened fine-grained ship recognition performance. In order to solve the spatial misalignment problem that occurs in FPN, a spatial-aligned FPN (SAFPN) is investigated. SAFPN employs a spatial-aware alignment fusion module (SAFM) to effectively extract rich fine-grained features between adjacent-scale feature maps. Moreover, in response to the challenge posed by low category separability in features due to the intraclass diversity and interclass similarity among ship targets, an adaptive prototypical contrastive learning (APCL) method is further proposed. By introducing prototypical contrastive loss, APCL effectively enhances the category separability of ship features, thereby improving the performance of fine-grained ship recognition. Numerous experiments are validated on two fine-grained ship recognition datasets: FGSD and ShipRSImageNet. The experimental results demonstrate that the proposed SAFPN and APCL facilitate the model in extracting fine-grained features with strong category separability, effectively enhancing the performance of fine-grained ship recognition. Our code will be public and available athttps://github.com/liyangfan0/Fine-Grained-Ship-Recognition.

Deep learning can effectively extract latent information from data to enhance target-background separation in hyperspectral target detection (HTD). However, these models typically require extensive labeled samples, while available target spectra in hyperspectral images (HSI) are scarce. Additionally, existing deep models struggle with target detection in complex backgrounds due to subtle spectral differences. To address these issues, we propose a novel HTD method based on diffusion model and convolutional gated linear unit (HTD-DMCG). First, the diffusion model is integrated with MixUp for data augmentation to generate a diverse and sufficiently large sample set. Next, a Transformer architecture utilizing a convolutional gated linear unit is designed to effectively capture global dependencies and local feature correlations, leading to more discriminative feature representations. Additionally, a new target aggregation and background separation loss is introduced, which emphasizes target sample aggregation while increasing the distance between targets and background samples to enhance separability. The HTD-DMCG method is compared against classical and state-of-the-art HTD methods on four real HSI datasets. Extensive experiments show that it can effectively outperform existing methods in target detection performance. The code is available at https://github.com/Li-ZK/HTD-DMCG.

Cross-modal remote sensing image matching serves as a key technique for collaborative utilization of multi-source information. However, modal differences and geometric distortions between multi-source images pose challenges to existing methods in terms of robustness and generalization. To achieve feature interaction in cross-modal scenes, this paper proposes SwinMatcher, an end-to-end matching model based on the Transformer architecture. Innovatively proposing the window/shifted-window cross-attention based on the window/shifted-window self-attention mechanisms of Swin Transformer, SwinMatcher enables efficient cross-modal feature interaction and multi-scale contextual modeling. It also incorporates a learnable matching module to directly generate semi-dense correspondences. Moreover, a cross-modal remote sensing image matching dataset is generated, which encompasses four modalities: visible light, synthetic aperture radar (SAR), light detection and ranging (LiDAR), and map, distributed across four representative scenes. The dataset includes 400 samples produced via random homography transformations, designed to enhance modal diversity and scene complexity. Experiments demonstrate that SwinMatcher outperforms state-of-the-art methods on this new dataset as well as public benchmarks, exhibiting superior robustness under complex scenes involving coupled modal and geometric distortions. The proposed method and dataset provide novel solutions and evaluation benchmarks for cross-modal remote sensing image matching. The code and testing dataset will be made publicly available at https://github.com/LotrL/SwinMatcher.

With the advancement of point cloud technology, its significance in fields like remote sensing has escalated, especially in tasks like 3D terrain modeling and environmental monitoring. Except for positional information, point clouds also contain diverse information like color and intensity. Effectively utilizing these heterogeneous point features from large-scale point clouds to attain precise semantic segmentation is a considerable challenge. To alleviate the feature conflict problem in semantic segmentation, the Heterogeneous Feature Alignment Network (HFA-Net) is proposed. This method employs two network branches to align heterogeneous features and facilitate information exchange. Initially, a dual-branch architecture is designed to independently process heterogeneous features, aiming to extract unique knowledge from these features. Furthermore, a homogeneous feature aggregation module is established to aggregate features from the decoders of both network branches into a unified space. To enhance feature discrimination, constraints are imposed within this module. These constraints aim to reduce the distance between features of the same class while increasing the distance between those of different classes, thereby enhancing semantic-level information exchange. Additionally, a pointwise alignment module is proposed to enhance the similarity between pointwise predictions generated by the dual-branch network, promoting information exchange at the prediction level. Experimental results, obtained by integrating the benchmark network for point cloud semantic segmentation into our proposed framework, reveal that the network trained within this framework exhibits superior performance compared to existing network.

In recent years, unmanned aerial vehicle (UAV) technology has shown great potential for application in hyperspectral image (HSI) classification tasks due to its advantages of flexible scheduling and fast response. However, existing deep learning-based classification algorithms have not thoroughly studied the problems caused by the higher spatial resolution of UAV HSIs. Such as more severe intra-class variation and higher computational resource consumption. These issues limit classification performance. To address these challenges, this paper proposes a lightweight spatial-spectral deformable convolutional neural network (LS2DCNet) for UAV HSI classification. This network reduces computational resource usage and data processing time, meeting the fast-response application requirements while ensuring classification performance. First, spatial-spectral deformable convolution (S2DConv) is designed to construct a lightweight feature extraction network. This not only enhances the adaptive extraction of fine features but also reduces computational resource consumption, and improves response speed. Second, a dynamic labeling-based joint loss (DL-JLoss) is designed to dynamically learn the distribution relationship between data classes. This improves the network’s generalization performance. A more realistic experimental validation was conducted on three UAV HSI datasets using regionally divided training samples, and inference comparisons were performed on embedded devices. The results show that the proposed LS2DCNet exhibits a better overall performance in terms of classification accuracy and inference speed.The relevant code can be found at https://github.com/niuroushu/A-Lightweight-Spatial-Spectral-Deformable-CNN-for-UAV-Hyperspectral-Image-Classification.

Collaboratively utilizing the complementary information provided by hyperspectral imagery and light detection and ranging (LiDAR) data will extend the applications associated with land cover recognition and mapping. Existing joint classification algorithms mainly focus on learning complementary patterns in the pure spatial domain, while paying little attention to complementary cues in the spatial-frequency domain. The model’s expressive capability of these methods may be limited by an upper bound subject to the spatial domain. To fill this gap, a Dynamic Multiple Fractional Fourier Domains Fusion (DMFraF) is proposed for joint classification of hyperspectral and LiDAR data. Firstly, to comprehensively learn the complementary patterns between HSI and LiDAR data, we transform the features of two modalities into multiple fractional domains containing different spatial-frequency components for multimodal fusion. Secondly, to obtain the optimal representation from the multimodal features of multiple fractional domains, we propose a dynamic fusion scheme guided by the optimal transport (OT) technique, which can dynamically adjust the contributions from different fractional domains. Finally, to extract purer modality-specific features, we propose a channel aggregation Transformer encoder with central cross-attention (C2AT encoder), to aggregate channel-wise features of central pixels into the spatial branch and compress interference from noisy surroundings. Extensive experiments and analysis on three hyperspectral and LiDAR datasets suggest the superiority of the proposed method.

In recent years, language-supervised vision models have demonstrated impressive potential in learning open-world concepts. Some research has introduced this learning paradigm to the hyperspectral image (HSI) processing domain; however, there has been limited work integrating textual information into the hyperspectral open-set recognition task. To fill this gap, we leverage textual supervision information in open-set HSI classification (HSIC) and propose a language-enhanced dual-level contrastive learning network (LDCLNet). Specifically, we introduce a linguistic mode with prior knowledge as a supervised signal to enhance the metric distances between closed-set samples and provide supplementary semantic information for open-set samples. Second, a dual-level visual-language (V-L) contrastive learning (CL) approach, which can align visual and language embeddings separately at the instance level and manifold level, is proposed to establish a more accurate link between visual and language representations. Finally, a distance-refined open-set recognition method is proposed, which aims to effectively discover unknown class samples during testing by refining predictions of known and unknown classes. Extensive experiments and analysis on three public HSI datasets validate the effectiveness of LDCLNet.

Deep learning has garnered extensive attention in hyperspectral image (HSI) processing. However, its application in HSI semantic segmentation tasks has been relatively limited. Although segmentation methods can often interpret images up to two orders of magnitude faster than classification methods when interpreting images of the same scene, the segmentation task requires the training data to be fully labeled, i.e., each pixel has a corresponding label. Such data are scarce in HSI data. To address this problem, this article proposes a lightweight segmentation network based on a hierarchical attention segmentation network (LHAS), in which a generalized data augmentation (GDA) method is utilized to acquire relatively sufficient data for semantic segmentation. Specifically, the hierarchical attention module is designed to extract global and local information on HSI patches from different layers. A prototype auxiliary module (PAM) of cluster contrast has also been developed to enhance feature discrimination. Across two different datasets in various scenarios, the proposed LHAS demonstrates superior segmentation performance compared to existing methods, affirming its effectiveness. In addition, experiments conducted on embedded devices validate the efficacy of LHAS.

Convolutional Neural Networks (CNNs) have achieved remarkable breakthroughs on remote sensing tasks in recent years. However, deploying CNNs for real-time remote sensing on-board processing still remains a challenge due to power consumption, real-time and other limitations. Therefore, in this article, a satellite-based real-time remote sensing accelerator is proposed, where algorithm and hardware approaches are proposed to jointly optimize CNNs’ deployment on edge-side aerospace devices. Firstly, a collaborative-trainable sparse-quantization (CTSQ) method is proposed to reduce the model’s storage overhead. In the CTSQ method, analysis of the errors is performed for the sparsity-quantization composition. Besides, the inter-channel correlations among parameters are leveraged, where the structured sparsity and quantization are performed with fine-grained units. Secondly, a modular-system co-optimized (MoSyC) architecture is proposed. A hardware-mapped sparse access (HMSA) strategy is proposed to effectively filter out zero elements in sparse parameters. Moreover, a high-throughput architecture is designed for parallel and pipelined data flow control. Finally, extensive experiments are conducted on both scene classification and object detection tasks with ResNet and YOLOv5 models. The results show that the proposed CTSQ method achieves the compression ratio of more than 13.81 times, and the proposed MoSyC architecture achieves the throughput of more than 1815 GOPS, demonstrating the effectiveness of the proposed accelerator.

In recent years, reconstruction-based methods have achieved excellent detection results in the field of hyperspectral anomaly detection (HAD). These methods predominantly operate on two aspects regarding their working principles: 1) reconstructing background pixels and 2) suppressing anomalous pixels. However, most methods only tackle the HAD task from the spatial and spectral domains, making it challenging to effectively suppress anomalies. To eliminate these issues, this article proposes a Fourier conditional mask-based mixed attention (FCMMA) method. First, we propose the FCMMA method for HAD. FCMMA generates a conditional mask (CMASK) that suppresses anomalous high-frequency information and preserves background low-frequency information in the frequency domain, optimizing the anomaly detection process. In addition, to achieve fine-grained HAD, we propose the Fourier anomaly suppression filter (FASF). FASF uses Fourier techniques to manage background and anomalies, improving detection via precise frequency decoupling. Finally, a CMASK network is designed to effectively suppress anomalies. The CMASK network integrated the FASF module and the spatial-spectral multilayer perceptual (SSMLP) machine module together to enhance the transformation and representation capabilities of the generated masks, which can also help suppress anomalies. The results on five different datasets show that the proposed method is more effective and superior when compared to nine state-of-the-art methods.

Displacement estimation using remote sensing images is an effective approach for assessing surface displacement caused by natural disasters like earthquakes and landslides. By employing pixel correlation algorithms, high-precision displacement maps can be generated from images taken before and after surface movement. However, traditional methods often rely on spatial regularization or frequency masking to reduce high-frequency noise, which can smooth spatial details and result in biased displacement estimates, especially near sharp discontinuities typical of earthquake surface ruptures. Moreover, sub-pixel displacement estimation using Synthetic Aperture Radar (SAR) images remains a challenge compared to optical images, due to the strong impact of speckle noise. This paper presents GeoFlowNet-SAR, an innovative sub-pixel displacement estimation method leveraging SAR images. SAR offers advantages thanks to an all-weather observation and high penetration, making it suitable for conditions typically challenging for optical systems in the visible light spectrum. This study uses Sentinel-1 SAR Single Look Complex (SLC) images with dual-polarization (VV and VH modes) and Interferometric Wide (IW) swath mode to balance coverage and resolution. By training on simulated displacement datasets with realistic sharp discontinuities, GeoFlowNet-SAR directly predicts surface displacement fields, providing highly efficient, robust, and precise results, while overcoming some limitations of traditional methods. The effectiveness of the proposed methodological contribution is first quantitatively demonstrated using synthetic simulated earthquake datasets, including comparisons with state-of-the-art correlation methods. The method is further validated using two real remote sensing images from the 2019 Ridgecrest earthquake and from the 2023 Turkey-Syria earthquake. The observed results from these real datasets confirm the effectiveness of GeoFlowNet-SAR in practical applications. The codes are available at: https://gricad-gitlab.univ-grenoble-alpes.fr/giffards/geoflownet-sar.

Remote sensing data from real-world scenarios manifests a long-tailed distribution, with the continuous emergence of new classes over time. Nevertheless, the existing class-incremental remote sensing classification models neglect the above long-tailed distribution phenomenon, which seriously damages their overall superior performance. Meanwhile, long-tail class-incremental learning developed in other areas focuses only on the classifier decision boundary optimization of the tail-class, while neglecting the robustness of the feature backbone. The feature backbone trained on the base classes causes a serious significant distribution shift for the incremental classes owing to the distributional differences between base and incremental classes. To solve these issues, we propose a continually evolved feature and classifiers learning (CEF-CL) framework for long-tail class-incremental remote sensing scene classification. Specifically, tail-class data are scaled and grafted onto head-class data to diversify the semantic information of the tail-class leveraging the rich context of the head classes, which can improve the generalization of the feature backbone. And then, an adaptive multi-scale feature fusion (AMFF) module is proposed to couple feature maps of head and tail classes scale by scale for generating virtual tail-class features that deeply perceive head-class information, which can further enhance the reliability of classifier decision boundary optimization. Furthermore, examples from old classes are regarded as pseudo-tail classes to participate in incremental learning, which greatly alleviates catastrophic forgetting of old classes. Extensive experiments on two remote sensing benchmarks demonstrate the superiority of the proposed CEF-CL in comparison with existing class-incremental learning.

To address the challenges posed by extensive modality differences in image matching tasks, this paper proposes a cascaded learning framework. It guides the optimization of an end-to-end matching model via a dynamic data engine, which can provide sufficient cross-modal training data to support the model’s full adaptation to cross-modal features. The data engine integrates a random homography transformation module and a lightweight image generation model, enabling the online synthesis of cross-modal image pairs with geometric variations and diverse styles. This provides the matching model with rich cross-modal stimulation. The matching model adopts a hybrid architecture combining a CNN backbone and Transformer attention mechanisms, which integrates multi-scale local feature extraction with global context modeling. By adopting the proposed stepwise aggregation strategy, the efficiency of feature extraction is well ensured. Subsequently, a coarse-to-fine matching strategy is employed to achieve high accuracy and robustness of feature alignment. Comprehensive experiments on both self-collected and public cross-modal image matching datasets demonstrate that the proposed DGIM outperforms existing state-of-the-art approaches in cross-modal matching performance while achieving a good balance between efficiency and effectiveness. It also exhibits broad practical potential across multiple fields and scenes. This work provides novel solutions and evaluation benchmarks for cross-modal image matching tasks. The code and testing dataset will be made publicly available at https://github.com/LotrL/DGIM.

Deep learning models exhibit promising potential in multimodal remote sensing image semantic segmentation (MRSISS). However, the constrained access to labeled samples for training deep learning networks significantly influences the performance of these models. To address that, self-supervised learning (SSL) methods have garnered significant interest in the remote sensing community. Accordingly, this article proposes a novel multimodal contrastive learning framework based on tuple perturbation, which includes the pretraining and fine-tuning stages. First, a tuple perturbation-based multimodal contrastive learning network (TMCNet) is designed to better explore shared and different feature representations across modalities during the pretraining stage and the tuple perturbation module is introduced to improve the network’s ability to extract multimodal features by generating more complex negative samples. In the fine-tuning stage, we develop a simple and effective multimodal semantic segmentation network (MSSNet), which can reduce noise by using complementary information from various modalities to integrate multimodal features more effectively, resulting in better semantic segmentation performance. Extensive experiments have been carried out on two published multimodal image datasets including optical and synthetic aperture radar (SAR) pairs, and the results show that the proposed framework can obtain more superior performance of semantic segmentation than the current state-of-the-art methods in cases of limited labeled samples. The source code is available athttps://github.com/yeyuanxin110/TMCNet-MSSNet.

Due to recent advancements in deep learning, techniques for urban structure extraction and semantic segmentation of multimodal remote sensing images have significant improvements. However, the challenge arises from the variable color intensity and complex texture of urban structures in optical images, particularly in buildings and roads. Fortunately, the light detection and ranging (LiDAR) images promote the task of developing an optimal multimodal fusion network that effectively leverages information from different modalities. In this article, we propose an attention multiscale network (AMSNet) for binary semantic segmentation tasks focused on building extraction, as well as multiclass semantic segmentation tasks, by integrating optical and LiDAR remote sensing images. AMSNet introduces two feature fusion modules—spatial scale adaptive fusion (S2AF) and semantic guided fusion (SGF). S2AF facilitates feature fusion between optical and LiDAR images within the same layer. This module contains a spatial scale selection strategy and an adaptive weight learning strategy, which enables the network to adaptively extract and intentionally select multiscale features from multimodal data. SGF addresses the semantic gap between different layered block features through semantic feature guidance strategy while achieving feature fusion. Furthermore, we introduce robust feature learning (RFL) to ensure the network robustness in rotation and variation in objects, making it resilient to images captured from different viewpoints and sensors. RFL incorporates point-to-point similarity learning strategy and multiscale feature reuse strategy. Experimental results on publicly available datasets demonstrate that AMSNet outperforms other state-of-the-art models. Extensive ablation studies further confirm the significance of all key components in the proposed approach. The source code of this method is available athttps://github.com/B-LG-J/AMSNet.git.

Wetlands are critical ecosystems where fine-grained land cover classification is crucial for effective management. Although UAV-based hyperspectral imaging provides both high spectral and spatial resolution, we observed significant confusion among spectrally and spatially similar classes in our self-collected Yellow River wetland dataset. To address this, we propose the Spectral-Spatial Multidimensional Collaborative Network (SS-MCNet), which comprises a Spectral Probe Module (SPM) for fine spectral representation, a Spatial Texture Module (STM) for detailed texture extraction, and a Spectral-Spatial Interaction Module (SSIM) to model cross-domain dependencies. A Collaborative Attention Fusion mechanism is introduced for adaptive balancing of spectral and spatial information, and a consistency-diversity loss function is designed to maintain consistency and complementarity among the three modules. Experimental results show that SS-MCNet achieves state-of-the-art performance with 98.08% overall accuracy (OA), 97.42% average accuracy (AA), and a Kappa coefficient of 97.75%, and performs well in an additional hyperspectral–panchromatic fusion experiment, providing a reference for multi-source fusion classification.

The joint application of multisource remote sensing (MSRS) data, such as hyperspectral image (HSI) and light detection and ranging (LiDAR), offers significant potential for accurate land cover classification. However, the existing applications often struggle with domain shifts across scenes caused by sensor, illumination, and phase variations. Focusing on this domain adaptation problem, a Chirplet Fourier analysis network (ChirpFAN) is proposed for cross-scene classification of MSRS data in this paper. Firstly, a fractional spatial-frequency-phase feature extraction module including the fractional Fourier transform and a learnable phase-aware weighting block is proposed to capture multi-domain features. Secondly, a Chirplet swin transformer (ChirpST) block integrates a Chirplet Fourier analysis (ChirpFA) layer within a Swin transformer is designed to analyze multi-scale textural and oscillatory patterns. Finally, a modality-shared network including ChirpST blocks is designed for inter-modal fusion and alignment. Extensive experiments demonstrate that the ChirpFAN framework achieves state-of-the-art performance with 3% average improvements on three challenging cross-scene MSRS datasets. Code will be released on GitHub.

In recent years, deep learning has shown immense promise in advancing medical hyperspectral imaging diagnostics at the microscopic level. Despite this progress, most existing research models remain constrained to single-task or single-scene applications, lacking robust collaborative interpretation of microscopic hyperspectral features and spatial information, thereby failing to fully explore the clinical value of hyperspectral data. In this paper, we propose a microscopic hyperspectral universal feature perception framework (UFPF), which extracts high-quality spatial-spectral features of hyperspectral data, providing a robust feature foundation for downstream tasks. Specifically, this innovative framework captures different sequential spatial nearest-neighbor relationships through a hierarchical corner-to-center mamba structure. It incorporates the concept of "progressive focus towards the center", starting by emphasizing edge information and gradually refining attention from the edges towards the center. This approach effectively integrates richer spatial-spectral information, boosting the model's feature extraction capability. On this basis, a dual-path spatial-spectral joint perception module is developed to achieve the complementarity of spatial and spectral information and fully explore the potential patterns in the data. In addition, a Mamba-attention Mix-alignment is designed to enhance the optimized alignment of deep semantic features. The experimental results on multiple datasets have shown that this framework significantly improves classification and segmentation performance, supporting the clinical application of medical hyperspectral data. The code is available at: https://github.com/Qugeryolo/UFPF.

Hyperspectral imaging technology is considered a new paradigm for high-precision pathological image segmentation due to its ability to obtain spatial and spectral information of the detected object simultaneously. However, due to the time-consuming and laborious manual annotation, precise annotation of medical hyperspectral images is difficult to obtain. Therefore, there is an urgent need for a semi-supervised learning framework that can fully utilize unlabeled data for medical hyperspectral image segmentation. In this work, we propose an adversarial consistency constraint learning cross indication network (ACCL-CINet), which achieves accurate pathological image segmentation through adversarial consistency constraint learning training strategies. The ACCL-CINet comprises a contextual and structural encoder to form the spatial-spectral feature encoding part. The contextual and structural indications are aggregated into features through a cross indication attention module and finally decoded by a pixel decoder to generate prediction results. For the semi-supervised training strategy, a pixel perceptual consistency module encourages the two models to generate consistent and low-entropy predictions. Secondly, a pixel maximum neighborhood probability adversarial constraint strategy is designed, which produces high-quality pseudo labels for cross supervision training. The proposed ACCL-CINet has been rigorously evaluated on both public and private datasets, with experimental results demonstrating that it outperforms state-of-the-art semi-supervised methods. The code is available at: https://github.com/Qugeryolo/ACCL-CINet.

Hyperspectral imaging is endowed with outstanding discriminability between different land types by its comprehensive sensing of the spectrum, thus favored applying to anomaly detection. However, blurring effect, as a critical cause for quality deterioration of hyperspectral imaging, has been omitted by previous hyperspectral anomaly detection models. On one hand, given that anomalies are sparsely distributed in nature, such blurring effect entangling neighboring pixels severely weighs those detection models down. On the other hand, abnormal objects jeopardize the low-dimensional structure of the image, thus deblurring those images with anomalies is more challenging than normal ones. Hence, it is of much significance to investigate anomaly detection using blurred hyperspectral images. To this end, this paper proposes a generalized non-convex surrogated tensor framework that is able to perform anomaly detection robustly to blurring effects on hyperspectral images. The proposed framework is featured to be a unified paradigm which guarantees convergence for a broad class of non-convex surrogates. Through treating the spatial and spectral low-rankness adaptively via Block Term Decomposition, the unevenness in the multi-linear low-rankness of hyperspectral image is comprehensively considered, which together with the non-convex surrogates results in a tighter modeling of the low-dimensional prior of hyperspectral images. Extensive experiments demonstrate the superiority of the proposed method compared with the state-of-the-art methods on both hyperspectral image deblurring and anomaly detection.

Hyperspectral video (HSV) offers valuable spatial, spectral, and temporal information simultaneously, making it highly suitable for handling challenges such as background clutter and visual similarity in object tracking. However, existing methods primarily focus on band regrouping and rely on RGB trackers for feature extraction, resulting in limited exploration of spectral information and difficulties in achieving complementary representations of object features. In this paper, a spatial-spectral fusion network with spectral angle awareness (SSF-Net) is proposed for hyperspectral (HS) object tracking. Firstly, to address the issue of insufficient spectral feature extraction in existing networks, a spatial-spectral feature backbone ( $S^{2}$ FB) is designed. With the spatial and spectral extraction branch, a joint representation of texture and spectrum is obtained. Secondly, a spectral attention fusion module (SAFM) is presented to capture the intra- and inter-modality correlation to obtain the fused features from the HS and RGB modalities. It can incorporate the visual information into the HS context to form a robust representation. Thirdly, to ensure a more accurate response to the object position, a spectral angle awareness module (SAAM) is designed to investigate the region-level spectral similarity between the template and search images during the prediction stage. Furthermore, a novel spectral angle awareness loss (SAAL) is developed to offer guidance for the SAAM based on similar regions. Finally, to obtain the robust tracking results, a weighted prediction method is considered to combine the HS and RGB predicted motions of objects to leverage the strengths of each modality. Extensive experiments on the HOTC-2020, HOTC-2024, and BihoT datasets demonstrate the effectiveness of the proposed SSF-Net compared with state-of-the-art trackers. The source code will be available at https://github.com/hzwyhc/hsvt.

To address the core issue of high-dimensional data processing in hyperspectral pathological diagnosis, we develop a new feature selection framework using functional data analysis (FSFDA). The framework models pixel spectra as continuous functions to preserve spectral continuity, overcoming the limitations of traditional discrete representations. Based on functional features, an innovative adaptive spectral segmentation strategy driven by functional change rate is developed to achieve optimal segmentation in the feature space. Additionally, a multi-criteria scoring mechanism including supervised (FSFDA-S) and unsupervised (FSFDA-U) paradigms is developed to enhance feature diagnostic discriminability while maintaining sparsity. Experimental results on the pathological hyperspectral image dataset of membranous nephropathy validate that the proposed method achieves over 99% classification accuracy while reducing feature dimensions by 94.5%. For cross-modal data involving in-vivo human brain and white blood cells, FSFDA effectively identifies diagnostic bands aligned with histopathological signatures, verifying its adaptive feature selection ability and cross sample generalization performance.

Hyperspectral image change detection (HSI-CD) possesses strong capabilities in exploring subtle changes in land cover. Due to sensor noise and imaging conditions, different semantic land covers in the same spatial location may exhibit similar spectral characteristics, leading to pseudoinvariant phenomena (identification of changed areas as unchanged areas) and causing a higher rate of false negatives in the model. Existing methods primarily focus on obtaining auxiliary discriminative information from spatial correlations or temporal dependencies. However, the frequency domain, which possesses rich global gradient distribution information, is often overlooked. The fractional Fourier transform (FrFT) is an extension of the Fourier transform (FT), representing a temporal-frequency local transformation suitable for processing nonstationary signals. Furthermore, multiorder fractional Fourier domains provide more observable domains for change discrimination. In this work, the application of FrFT is extended to the field of HSI-CD, and an adaptive weighted metric learning network based on fractional domain decoupling (FrFTML) is proposed. Specifically, the fractional domain decoupling (FrDD) module transforms the original HSI into multiorder FrFT domains and extracts their rich spatial-frequency mixed information, effectively suppressing noise while enhancing the representation of subtle differences. In addition, an adaptive weighted metric learning (AWML) framework is designed to merge multiorder fractional Fourier domain information in an adaptively weighted fusion manner. It introduces deep metric learning to explore the distances between samples of different categories that have relatively high similarity, so as to guide the direction of adaptive weighted fusion. Finally, the differential mask attention (DMA) module is designed to explore global contextual differences between bitemporal HSIs, obtaining change features with well-represented differences. Some experiments conducted on three public datasets indicate that FrFTML outperforms other state-of-the-art methods. Furthermore, the proposed method exhibits superiority in dealing with land cover that may lead to pseudoinvariant phenomena (identification of changed areas as unchanged areas).

The availability of multisource remote sensing data provides the possibility for comprehensive observation. Convolutional neural networks (CNNs) naturally integrate multisource feature extractors and classifiers into an end-to-end multilayer design. However, CNN assumes data are independent and identically distributed. In practice, it is not always possible to access the labels or even data of the testing scenes. Therefore, the CNN-based methods have exposed its limitation on generalization ability. To solve the issue, a feature-distribution-independent network (FDINet) is designed for multisource remote sensing cross-domain classification without feature alignment and decoupling operations. On one hand, an elegantly designed baseline is used for extracting multisource cross-domain features. The baseline extracts the common line and texture features through shallow weight-sharing networks. More importantly, the modality prediction probability is used to measure the similarity between the source domains and the target domains, thereby improving cross-domain collaboration capabilities. On the other hand, the sharpness-aware feature discriminating (SAFD) strategy is developed for model optimization. Specifically, the generalization ability is improved by minimizing the sharpness of local optima. To avoid the decrease in feature discrimination caused by the gradient conflict between sharpness and overall loss, the discrimination constraints are designed to balance feature discrimination and generalization ability. Comprehensive experiments are conducted on two datasets, which demonstrate that the proposed FDINet outperforms other competitors in terms of quantitative and qualitative analyses.

Multisource remote sensing data classification is a challenging research topic, and how to address the inherent heterogeneity between multimodal data while exploring their complementarity is crucial. Existing deep learning models usually directly adopt feature-level fusion designs, most of which, however, fail to overcome the impact of heterogeneity, limiting their performance. As such, a multimodal joint classification framework, called global clue-guided cross-memory quaternion transformer network (GCCQTNet), is proposed for multisource data [i.e., hyperspectral image (HSI) and synthetic aperture radar (SAR)/light detection and ranging (LiDAR)] classification. First, a three-branch structure is built to extract the local and global features, where an independent squeeze-expansion-like fusion (ISEF) structure is designed to update the local and global representations by considering the global information as an agent, suppressing the negative impact of multimodal heterogeneity layer by layer. A cross-memory quaternion transformer (CMQT) structure is further constructed to model the complex inner relationships between the intramodality and intermodality features to capture more discriminative fusion features that fully characterize multimodal complementarity. Finally, a cross-modality comparative learning (CMCL) structure is developed to impose the consistency constraint on global information learning, which, in conjunction with a classification head, is used to guide the end-to-end training of GCCQTNet. Extensive experiments on three public multisource remote sensing datasets illustrate the superiority of our GCCQTNet with regards to other state-of-the-art methods.

Arbitrary-oriented object detection (AOOD) has been widely applied to locate and classify objects with diverse orientations in remote sensing images. However, the inconsistent features for the localization and classification tasks in AOOD models may lead to ambiguity and low-quality object predictions, which constrains the detection performance. In this article, an AOOD method called task-wise sampling convolutions (TS-Conv) is proposed. TS-Conv adaptively samples task-wise features from respective sensitive regions and maps these features together in alignment to guide a dynamic label assignment for better predictions. Specifically, sampling positions of the localization convolution in TS-Conv are supervised by the oriented bounding box (OBB) prediction associated with spatial coordinates, while sampling positions and convolutional kernel of the classification convolution are designed to be adaptively adjusted according to different orientations for improving the orientation robustness of features. Furthermore, a dynamic task-consistent-aware label assignment (DTLA) strategy is developed to select optimal candidate positions and assign labels dynamically according to ranked task-aware scores obtained from TS-Conv. Extensive experiments on several public datasets covering multiple scenes, multimodal images, and multiple categories of objects demonstrate the effectiveness, scalability, and superior performance of the proposed TS-Conv.

Cross-scene hyperspectral image classification (HSIC) poses a significant challenge in recognizing hyperspectral images (HSIs) from different domains. The current mainstream approaches based on domain adaptation (DA) methods need to access target data when aligning distributions between domains, limiting the applicability of the model. In contrast, recent domain generalization (DG) methods aim to directly generalize to unseen domains, eliminating the requirements for target data during training. Nonetheless, most DG-based methods overly focus on randomizing sample styles, leading to semantically compromised samples. In addition, broadening the source distribution without ensuring reasonable support may result in undesired extended distributions. To address these issues, we propose a novel DG network with frequency disentanglement and data geometry (FDGNet) for cross-scene HSIC. Specifically, we first develop a spectral-spatial encoder based on frequency disentanglement (FDSS encoder), which facilitates synthesized domains to preserve their semantic consistency while simulating interdomain gaps with the source domain. Second, to avoid the generation of unrealistic samples, we incorporate data geometry into adversarial training. This helps diversify new domains while keeping the data geometry of extended domains in an explainable support. To improve the learning of domain-invariant representation, we propose an intermediate domain sampling strategy based on the class-wise perceptual manifold. This strategy synthesizes reliable intermediate domains by sampling from class-wise manifold flows estimated over the source and extended domains. Extensive experiments and analysis on three public HSI datasets yield the superiority of our proposed FDGNet. The codes will be available from the website: https://github.com/Qba-heu/FDGNet.

Hyperspectral object tracking (HOT) has many important applications, particularly in scenes where objects are camouflaged. The existing trackers can effectively retrieve objects via band regrouping because of the bias in the existing HOT datasets, where most objects tend to have distinguishing visual appearances rather than spectral characteristics. This bias allows a tracker to directly use the visual features obtained from the false-color images generated by hyperspectral images (HSIs) without extracting spectral features. To tackle this bias, the tracker should focus on the spectral information when object appearance is unreliable. Thus, we provide a new task called hyperspectral camouflaged object tracking (HCOT) and meticulously construct a large-scale HCOT dataset, BihoT, consisting of 41912 HSIs covering 49 video sequences. The dataset covers various artificial camouflage scenes, where objects have similar appearances, diverse spectrums, and frequent occlusion (OCC), making it a challenging dataset for HCOT. Besides, a simple but effective baseline model, named spectral prompt-based distractor-aware network (SPDAN), is proposed, comprising a spectral embedding network (SEN), a spectral prompt-based backbone network (SPBN), and a distractor-aware module (DAM). Specifically, the SEN extracts spectral-spatial features via 3-D and 2-D convolutions to form a refined prompt representation. Then, the SPBN fine-tunes powerful RGB trackers with spectral prompts and alleviates the insufficiency of training samples. Moreover, the DAM utilizes a novel statistic to capture the distractor caused by occlusion from objects and background and corrects the deterioration of the tracking performance via a novel motion predictor. Extensive experiments demonstrate that our proposed SPDAN achieves the state-of-the-art performance on the proposed BihoT and other HOT datasets.

Most of the existing fusion algorithms are not robust to unregistered input images. Even after image registration, nonlinear nonregistration may persist in the local areas of the images, leading to poor quality in the fused image. So, as to tackle these challenges, a progressive remote sensing image registration and fusion network is proposed in this article, and named PRF-Net, which is particularly useful when two images are from different platforms. First, a registration network is designed to register the input image patches, which includes a global spatial transform network (GSTN) and a local spatial warp network (LSWN). The GSTN is primarily used for coarse registration, applying rigid transformation to globally align the input images. After coarse registration, the preliminarily registered moving image is input into the LSWN for local fine-tuning to maximize correlation between the input image patches. Subsequently, the fine registered images are degraded and input into the fusion network to generate the fused image. To maintain sufficient spectral and spatial information of the fused image, a multiscale feature extraction (MSFE) block with a highly interpretable spatial details attention (SDA) block is designed, which can enhance the ability of fusion network to extract and preserve spatial details and spectral information. Three groups of experiments conducted on four types of remote sensing images give evidence of that the proposed PRF-Net exhibits excellent performance in both reduced and full resolutions, showcasing its outstanding registration and fusion quality.

Image fusion aims to combine the complementary features of different modalities to produce an informative fused image. Due to the different imaging mechanisms, information conflicts may arise from infrared and visible source images. Existing infrared and visible fusion methods are devoted to preserving the features of source images as much as possible. However, handling conflicting information is often overlooked. Thus, we leverage the powerful generative priors of diffusion and propose a dual-diffusion structure, termed DSFuse, to handle conflicting information and achieve feature fidelity during image fusion processing. Diffusion modules are introduced to guide the fusion network to understand the meaningful information of the source image easily. First, the fusion network is used to retain features in the fused image as much as possible. Then, diffusion modules are used to reconstruct source images from noise based on the output of the fusion network. Finally, feedback from the diffusion modules forces the fusion network to aggregate modality information to ensure fidelity; the high quality of the fusion result is also profitable for a better reconstruction of diffusion modules, forming a positive feedback loop. In addition, we release a new dataset for infrared/visible fusion to support the fusion network training and evaluation, named the multiscene infrared and visible (MSIV) images dataset. Extensive experiments demonstrate that DSFuse outperforms other state-of-the-art (SOTA) fusion methods.

Segment anything model (SAM), as a prompt-based image segmentation foundation model, demonstrates strong task versatility and domain generalization (DG) capabilities, providing a new direction for solving cross-scene segmentation tasks. However, SAM still has limitations in multispectral cross-domain segmentation tasks, mainly reflected in: 1) insufficient information utilization, which is reflected in the neglect of nonvisible spectral information and the shift information contained in source domain (SD) samples and target domain (TD) samples; and 2) lack of cross-domain strategies, which leads to insufficient cross-domain adaptation (DA) ability in downstream tasks. To address these challenges, we combine the respective advantages of masked autoencoder (MAE) and cross-domain strategies, propose an improved SAM DA network structure called domain information mining and state-guided adaptation network (DSAnet), aiming to enhance SAM's performance in multispectral cross-domain segmentation tasks from both data and task levels. At the data level, DSAnet incorporates a style masking learning component, which randomly masks image features and replaces them with domain-specific learnable tokens, integrated with the image reconstruction task, to mine the style information and domain invariance of the image itself. At the task level, DSAnet introduces domain state learning and style-guided segmentation: domain state learning, through a state sequence modeling approach, designs specific state representations for SD and TD to capture interdomain differences, thereby reducing task shift. Meanwhile, the learned domain state information can be directly applied to the inference stage. Style prompt segmentation guides the segmentation training process of SD images with TD style prompts, improving SAM's adaptability in cross-domain multispectral segmentation downstream tasks. Extensive experiments on three multitemporal multispectral image (MSI) datasets demonstrate the superiority of the proposed method compared to state-of-the-art cross-domain strategies and SAM variant methods.

This paper aims at providing an effective multi-modal images invariant feature extraction and matching algorithm for the application of multi-source data analysis. Focusing on the differences and correlation of multi-modal images, a feature-based matching algorithm is implemented. The key technologies include phase congruency (PC) and Shi-Tomasi feature point for keypoints detection, LogGabor filter and a weighted partial main orientation map (WPMOM) for feature extraction, and a multi-scale process to deal with scale differences and optimize matching results. The experimental results on practical data from multiple sources prove that the algorithm has effective performances on multi-modal images, which achieves accurate spatial alignment, showing practical application value and good generalization. The codes are available at https://github.com/MrPingQi.

The changes in building are important basis for urban monitoring. However, due to the rarity and sparsity of the occurrence of changes in buildings, collecting effective bitemporal image pairs is challenging, as it requires long-term observation over several months or even years. Additionally, annotating large-scale change detection datasets is time-consuming and labor-intensive. Consequently, data scarcity issues lead to insufficient training of building change detection models. To address this, we propose a data generation method Building Generation GAN (BG-GAN). Different from other GANs, the BG-GAN is trained based on adversarial consistency loss, enabling the model to generate new bi-temporal image pairs with various types of building changes. To verify the effectiveness of the proposed methods on change detection task, BG-GAN is utilized to perform building change samples generation on two building change detection datasets (LEVIR-CD and WHU-CD). The experimental results demonstrate that the proposed method can improve the robustness and generalization of change detection model to detect pseudo changes.

With the development of hyperspectral imaging technology, the hyperspectral anomaly has attracted considerable attention due to its significant role in many applications. Hyperspectral images (HSIs) with two spatial dimensions and one spectral dimension are intrinsically three-order tensors. However, most of the existing anomaly detectors were designed after converting the 3-D HSI data into a matrix, which destroys the multidimension structure. To solve this problem, in this article, we propose a spatial invariant tensor self-representation (SITSR) hyperspectral anomaly detection algorithm, which is derived based on the tensor–tensor product (t-product) to preserve the multidimension structure and achieve a comprehensive description of the global correlation of HSIs. Specifically, we exploit the t-product to integrate spectral information and spatial information, and the background image of each band is represented as the sum of the t-product of all bands and their corresponding coefficients. Considering the directionality of the t-product, we utilize two tensor self-representation methods with different spatial modes to obtain a more balanced and informative model. To depict the global correlation of the background, we merge the unfolding matrices of two representative coefficients and constrain them to lie in a low-dimensional subspace. Moreover, the group sparsity of anomaly is characterized by$l_{2.1.1}$norm regularization to promote the separation of background and anomaly. Extensive experiments conducted on several real HSI datasets demonstrate the superiority of SITSR compared with state-of-the-art anomaly detectors.

Preprocessing procedures are commonly employed to reduce water-absorption bands and noise in hyperspectral images (HSIs). Nevertheless, they typically do not entirely eradicate noise. This is especially evident in scenarios that necessitate data of exceptional quality, such as cross-domain few-shot classification tasks. Within these specific conditions, the influence of remaining background noise on the ultimate results of classification is substantial. Furthermore, the presence of sample selection biases in the few-shot task might lead to the emergence of false statistical correlations between data from distinct domains, resulting in a decrease in the model’s ability to generalize. We propose a new method called spatial-spectral–semantic cross-domain few-shot learning (S3CFSL) to address the challenge. This method promotes the learning of transferable information by incorporating feature denoising operations in the feature extraction process to restore essential information. Concurrently, it enhances cross-domain distributional consistency by introducing a semantic-aware strategy to strengthen the association between cross-domain data and semantic information. Specifically, the spatial and spectral dual channels (SSDCs), in conjunction with the cross-spatial-spectral transformer (CSST), are designed as a feature extractor to acquire interactive spatial-spectral features. The feature-denoising operations can further acquiring transferable information from cross-domain features, thus facilitating meta-learning in both the source domain (SD) and the target domain (TD). Meanwhile, a semantic-enhanced domain alignment (SEDA) is designed to promote domain adaptation by using a semantic-aware strategy, which significantly enhances distributional consistency for cross-domain tasks. Our results exhibit exceptional classification efficacy in comparison to other state-of-the-art approaches on three public HSI datasets.

At present, hyperspectral image classification (HSIC) technology has been warmly concerned in all walks of life, especially in agriculture. However, existing classification methods operate under the closed-set assumption, which deviates from the real world with open properties. At the same time, there are more serious phenomena of different crops with similar spectrum and same crops with different spectrum in agricultural hyperspectral data, which is also a great challenge to existing methods. In this work, a fractional Fourier based frequency-spatial-spectral prototype network is proposed to address the challenges of open-set hyperspectral image classification in agricultural scenarios. Firstly, fractional Fourier transform is introduced into the network to combine the information in the frequency domain with the spatial-spectral information, so as to expand the difference between different classes on the premise of ensuring the similarity between classes. Then, the prototype learning strategy is introduced into the network to improve the feature recognition capability of the network through prototype loss. Finally, in order to break the stubbornly closed-set property of closed-set classification method, the open-set recognition module is proposed. The difference between the prototype vector and the feature vector is used to judge the unknown class. Experiments on three agricultural hyperspectral datasets show that this method can effectively identify unknown class without sacrificing the classification accuracy of closed-set, and has satisfactory classification performance.

As a significant means of Earth observation, change detection in high-resolution remote sensing images has received extensive attention. Nevertheless, the variability in imaging conditions introduces style discrepancies and a range of pseudochange regions between bitemporal image pairs. Furthermore, changing objects possess diverse morphological representations, which makes accurately identifying change areas and delineating their boundaries within complex object distributions increasingly difficult. In response to the aforementioned challenges, we propose the Fourier feature interaction and multiscale perception (FIMP) model for effective change detection. To mitigate the impact of style discrepancies, FIMP employs the Fourier transform to adaptively filter bitemporal features in the frequency domain while mining the optimized bitemporal features relevant to the change detection task. To enhance the ability to recognize multiscale changing objects, FIMP aggregates and emphasizes the change areas with the introduced temporal change enhancement module (TCEM). By utilizing the U-fusion change perception module (UCPM) to perform multilevel bidirectional fusion of change features at different scales, FIMP can further enhance the ability to delineate complex semantic change boundaries. Experiments on three public datasets show that our approach outperforms seven state-of-the-art methods.

Hyperspectral image classification (HSIC) has been a popular task in recent years. Even benefiting from the rapid development of deep neural networks (DNNs), there are still remaining intrinsic problems, including inadequate utilization of spatial-spectral information and insufficient labeled samples. The recent emergency of diffusion models (DMs) came to the fore because of their impressive refined image generation performance. DMs have been proven to not only can capture the underlying information of data through training the decoder of DMs, but also have more stable training than GANs while retaining even better performance. To better perceive and utilize spectral-spatial information while alleviating insufficient labeled samples simultaneously, we introduce the DM into HSIC from a data generation perspective. Specifically, we propose a stage-wise DM framework (SWDiff), dividing the HSIC task into three stages, including: pretrain the diffusion decoder with the hyperspectral image (HSI); generate new HSI cubes through the well-trained decoder to extra supply the original HSI set; and utilize the supplied dataset to train varied classifiers to obtain a better classification performance. Suitable pretraining could enable the decoder to acquire spatial-spectral information of the HSIs sufficiently via modeling spectral-spatial relationships across samples, leading to better utilization of spectral and spatial information of HSIs. Furthermore, the DM could provide the inference stage with spatial-spectral prior knowledge to ensure the feasibility and plausibility of the dataset complement, which could alleviate the insufficient labeled samples problem. Eventually, the classification stage will benefit from the first two stages.

Object detectors often struggle with accuracy and generalization when applied to aerial imagery, primarily due to the following challenges: 1) great scale variation of objects in aerial images: both extremely small and large objects are visible in the same image; and 2) an extreme imbalance of the training sample between positive and negative anchors: there are several positive ground truth (GT) anchors and an abundance of negative anchors. In this article, we propose a dynamic training sample selection network (DTSSNet) to solve the above-mentioned problems in two dimensions. An attention-enhanced feature module (AEFM) is proposed to enhance the basic features by focusing on both channel and semantic information related to targets. This module provides more valuable information for accurately classifying objects of different scales. To tackle the imbalance in training samples, this article implements a dynamic training sample selection (DTSS) module that divides the training samples based on GT information. This module dynamically selects samples, ensuring a more balanced representation of positive and negative anchors, leading to improved learning. Importantly, the combination of AEFM and DTSS does not introduce any additional computational costs. Experimental evaluations on the VisDrone2019-DET dataset demonstrate that DTSSNet outperforms base detectors and generic approaches. Furthermore, the effectiveness of DTSSNet is validated on the UAVDT benchmark dataset, where it achieves state-of-the-art performance.

Accurate and efficient ship tracking by geosynchronous orbit (GEO) satellites holds great significance for large-scale maritime surveillance. Nevertheless, ship tracking continues to grapple with a multitude of challenges as follows: 1) the targets are small and often obscured by cloud interference, leading to weakened features; 2) the contrasts between the ships and the background are relatively low, complicating the identification and tracking process; and 3) the frame-to-frame relative positioning accuracy is poor, posing difficulties in reflecting the actual movement trends of ships. In response to these challenges, we proposed TS-Track, a novel framework employing multilevel supervision paradigm to improve tracking performance. Initially, this framework restructured the tracking task into three key sub-modules: image enhancement, object tracking, and trajectory adjustment, inherently fostering a unified training protocol that naturally encompasses all components. Subsequently, a trajectory-based frame fusion strategy was proposed, utilizing consecutive three-frame images to enhance target features and produce consistent motion feature patterns; Last but not least, a trajectory adjustment network was developed to correct the position of ships during tracking, resulting in stable tracking trajectories, and reproduce the actual movement trends of ships. The experimental results on GaoFen-4 dataset validated that our method delivered a significant improvement in ship tracking and achieved state-of-the-art (SOTA) performance. Source codes are available athttps://github.com/KTqizhi/KTqizhi.github.io.

Due to the lack of color and texture information and the fuzzy boundary of infrared (IR) small targets, the pixel-level mask annotation process consumes a lot of manual cost and is difficult to achieve accurate annotation. To further reduce the annotation burden, we propose an IR small target mask generation algorithm based on single-point supervised multi-scale chain growth clustering (MCGC). The core of this work is the adaptive generation of IR small-target pseudo mask maps under the supervision of randomly given single-point labels, sequentially through the strategies of multi-scale chain growth, Euclidean coefficient decay, K-Means clustering, and eight-neighborhood clustering. On the four public datasets, ablation experiments, qualitative and quantitative comparison experiments demonstrate that the MCGC algorithm has an efficient and accurate IR small target pseudo mask generation capability, which can be adapted to different numbers, scales, shapes, and intensities of targets in complex backgrounds. In addition, IR-Labelmask, an IR small target mask annotation software designed based on the MCGC algorithm, is publicly available on kourenke/IR-Labelmask-software (github.com). To our knowledge, this is the first mask annotation software designed for IR small target.

Fine-grained classification is of significant importance in the field of remote sensing. However, obtaining valuable and rare target images is often a challenging task, giving rise to the few-shot fine-grained classification problem. In response to this challenge, various meta-learning approaches have been introduced, with the feature map reconstruction network emerging as a prominent method. Targets in remote sensing images exhibit arbitrary orientation, substantial inter-class similarity and intra-class diversity. Nevertheless, the conventional feature map reconstruction network exhibits subpar performance due to its inability to handle rotational variations. Moreover, it only reconstructs features from a single channel dimension of support features, neglecting the interplay between different dimensions and resulting in inaccurate reconstruction errors. To overcome the challenges of imprecise rotational variation features for reconstruction and inaccurate reconstruction errors, we propose a rotation-invariant feature map complementary reconstruction network (RIFCRN). The RIFCRN involves several key innovations. First, we introduce a novel rotation-invariant module (RIM) based on active rotating filters and oriented response pooling, enabling the extraction of rotation-invariant features for reconstruction. This modification enhances the suitability of the feature map reconstruction network for the few-shot fine-grained classification problem. Second, we put forward a novel feature map complementary reconstruction (CPR) method that calculates the complementary reconstruction errors (CRE) which effectively captures relationships among different feature map dimensions and results in more accurate reconstruction errors. Finally, extensive experiments have been conducted to validate the effectiveness of the proposed RIFCRN in addressing the few-shot fine-grained classification problem. The code will be available at https://github.com/liyangfan0/RIFCRN.

Hyperspectral images have excellent spectral combining capabilities and LiDAR images have fine stereooscopic elevation information. Therefore, the multi-modal fusion classification of hyperspectral and LiDAR images is inevitably improves the interpretation ability of remote sensing images. In recent years, the MLP-Mixer, an image processing network based on MLP, has flourished in the field of image processing. In this work, we propose an innovative HyperMLP network based on the deep learning framework MLP-Mixer architecture to address the lack of spatial feature construction capability and the locality of multi-modal feature fusion in naive networks. Specifically,(1) The adoption of unsupervised superpixel embedding provides additional shallow morphological spatial feature information for the network, reduces the pressure of the feature extraction network, and enhances feature discrimination capabilities. (2) The feature scrambling strategy improves the diversity of features and strengthens generalization of the network by enhancing interactions between different spatial features. (3) By implementing the bilateral modulation strategy, feature fusion is applied at every stage of the deep network, reducing semantic drift between features. On three fiducial remote sensing datasets, classification tests are performed on the proposed HyperMLP network to verify its performance, and the results are definitely impressive.

With the advancement of high-resolution remote sensing satellites, the tracking of high-value targets such as planes and ships within satellite videos has become imperative. In recent years, several object tracking methods designed for satellite videos based on correlation filters have been proposed. However, these traditional correlation filters typically identify the location with the highest response value on the response map as the target position. In the context of satellite videos, where targets are often very small and surrounded by numerous similar objects, depending only on response values to determine the target’s location can easily lead to interference from nearby objects, resulting in tracking failures. Moreover, targets frequently encounter occlusion during their motion, further complicating tracking tasks due to the absence of distinctive target appearance features and leading to the issue of model drift. To address these challenges, we propose a novel distractor-occlusion aware correlation particle filters. Instead of determining the position with the maximum response value, our method initially selects the top k response values from the response map, creating a pool of candidates. Subsequently, we introduce an innovative quality score, rooted in motion information and response scores related to the target, for each candidate. Finally, these quality scores are employed to filter the most suitable candidate. This novel distractor-aware module effectively equips our tracking method to perform well in the presence of distractors. Additionally, to handle occlusion, we integrate the occlusion-aware module into the correlation particle filters, improving the tracker’s performance in occluded scenarios. To ensure the effective collaboration of the distractor-aware module and the occlusion-aware module, we introduce the dual Kalman Filter method. Our comprehensive experiments conducted on the SatSOT datasets conclusively demonstrate the effectiveness and superiority of our proposed tracking method. The code will be available at https://github.com/liyangfan0/DOCPF.

Cloud cover leads to great loss of spatial details in wide-swath multispectral images, and thus significantly affects their application value. Wide-swath images are huge in size and are usually cropped into patches before thin cloud removal. In addition, wide-swath images contain a rich variety of types and shapes of thin clouds, with each patch containing different clouds. However, most of the existing methods and datasets were primarily designed for natural image dehazing. These datasets had limited types and shapes of clouds. When applied to remote sensing image thin cloud removal task, these methods are unable to remove various types of clouds and lead to severe cross-patch color difference. To address this problem, a DecloudNet based on cross-patch consistency supervision was proposed. First, a multiscale cloud perception block (MCPB) with multisize convolutional kernels was proposed to enhance the network’s capability to extract clouds feature of different sizes. Second, a cross-patch consistency supervision was designed to reduce the network’s inconsistent cloud removal strength in different patches and remove cross-patch color difference when processing wide-swath images. Finally, a thin cloud simulation method based on Perlin noise, domain warping, and atmospheric scattering model was proposed to construct a high-quality declouding dataset containing clouds of multiple sizes and shapes, which can improve the performance of DecloudNet on different kinds of thin clouds. The DecloudNet and compared methods were tested for simulated thin cloud removal performance on images from QuickBird (QB), GaoFen-2 (GF2), and WorldView-2 (WV2) satellites, and for real thin cloud removal performance on wide-swath images from GaoFen-1 Wide Field of View (GF-1 WFV), GaoFen-1 (GF-1), and Earth-Observing-1 (EO-1) satellites. The experimental results demonstrated that DecloudNet outperformed the existing state-of-the-art (SOTA) methods. DecloudNet and cross-patch consistency supervision made it possible to perform thin cloud removal on wide-swath images of large size on most GPUs without worrying about graphics memory limitation. The source code and dataset are available athttps://github.com/N1rv4n4/DecloudNetthelink.

Recently, metric-based few-shot learning (FSL) methods have achieved good performance in hyperspectral image (HSI) classification. However, existing methods suffer from two problems: over-reliance on image modality information leads to inaccurate prototype representation, where a prototype refers to the centroid of each class in the dataset, and the impact of redundant and noisy pixels on model discriminability is rarely considered. These problems result in insufficient discriminability of the model for the target domain. To address the above issues, we propose a cross-domain few-shot HSI classification framework with cross-modal alignment and supervised contrastive learning (CDFS-CASCL). It is well known that human visual learning greatly benefits from the input of various modal information such as vision, language and video. Inspired by the way humans abstract image class concepts in language form and understand the essence of classes, we perform cross-modal alignment (CA) between similar image and text prototypes, and use abstract text semantics to guide the model to learn semantic related features with good generalization ability in images, so as to improve the accuracy of image prototypes representation of the prototypes. In addition, through supervised contrastive learning (SCL) based on neighborhood pixel mask in the target domain, the enhanced sample features belonging to the same class are closer, while the enhanced sample features belonging to different classes are pulled further, enabling the model to learn mask-robust discriminative feature representations, suppressing the negative impact of redundant and noisy pixels, and improving the model’s discriminability. The experimental results demonstrate the superiority of the proposed CDFS-CASCL. The code is available at https://github.com/Li-ZK/CDFS-CASCL-2024.

Pine wilt disease (PWD) is one of the most prevalent pine trees diseases, resulting in both ecological and economic havoc. UAV remote sensing segmentation plays a crucial role in early identifying and preventing PWD. However, deep learning segmentation models customized for PWD identification in scenarios with complex backgrounds have not received extensive exploration. In this paper, we propose a novel UAV remote sensing segmentation model called Clusterformer with a conventional encoder-decoder structure. The encoder is comprised of the specially designed Cluster Transformer, which includes a cluster token mixer and a spatial-channel feed-forward network (SC-FFN). The cluster token mixer utilizes constructed clusters from the feature maps to represent pixels, thereby reducing redundant and interfering information. The SC-FFN extracts multi-scale spatial information through depth-wise convolutions and channel information through a multilayer perceptron in sequence. The decoder primarily consists of the specially designed D-Cluster Transformer. The token mixer of the D-Cluster Transformer employs constructed clusters from high-level decoded tokens to represent low-level encoded tokens without relying on traditional upsampling methods such as interpolation, transpose convolution, or patch expansion. Consequently, more robust and less redundant features from high-level decoded feature maps are transferred to low-level encoded feature maps. Experimental results on two PWD datasets demonstrate that Clusterformer outperforms existing state-of-the-art segmentation models. This confirms the effectiveness and efficiency of Clusterformer in PWD identification. Code is available at https://github.com/huanliu233/Clusterformer.

Convolutional neural network (CNN)-based hyperspectral image (HSI) classification models often exhibit high volume and complexity. This not only poses challenges in deploying them on mobile and embedded devices due to storage and power constraints but also introduces a dilemma between the growing demand for labeled samples and the high cost associated with manual labeling. To address these challenges, we propose an ultra-lightweight hybrid CNN based on redundancy removal (ULite-R2HCN), specifically designed for HSI classification in scenarios with limited samples. To reduce computational costs and enhance feature extraction effectiveness, we focus on optimizing the widely used depthwise convolution (DW-Conv) and pointwise convolution (PW-Conv) in the lightweight HSI classification model. For DW-Conv, we design a spatial convolution with redundancy removal (R2Spatial-Conv). This involves the design of multi-scale 3D convolution kernels with specific structures instead of 2D convolution kernels, aiming to reduce redundant convolution kernels and extract multi-scale spatial features. Simultaneously, for PW-Conv, we design a spectral convolution with redundancy removal (R2Spectral-Conv). This utilizes a “copy-splicing-grouping” structure to extract spectral features within arbitrary range intervals, effectively reducing redundant spectral extractions and capturing long-range spectral relationships. Numerous experiments have shown that the proposed ULite-R2HCN achieves higher classification accuracy with an ultra-light volume for a few training samples. In addition, sufficient ablation experiments also verified the advanced performance of the designed R2Spatial-Conv and R2Spectral-Conv.

Existing hyperspectral cross-domain few-shot learning (FSL) methods focus mainly on elaborating on training strategies or domain alignment algorithms, while paying less attention to the biased meta-knowledge introduced by a large amount of source data and the implicit encouragement of learning target domain-specific attributes. In this paper, from the perspective of disentangled representation learning, a novel cross-domain FSL method based on feature disentanglement (FDFSL) is proposed for hyperspectral image classification (HSIC). Specifically, to suppress the representation biased towards the source data and enable the model to implicitly focus on the inherent knowledge of the target domain, an orthogonal low-rank feature disentanglement method is employed to acquire desired features of source and target pipelines. Furthermore, to preserve more shared and discriminative information from the heterogeneous data space (i.e., the spectral dimensions of the source and target scenes are typically different), a multi-order spectral interaction block based on central position encoding (MICD) is proposed to fully integrate the respective features into the spectral domain, which allows the model to emphasize informative spectral dimensions in a data-driven manner. Finally, to diversify the feature representation space while preventing the model overfitting domain alignment task, a self-distillation scheme is developed to facilitate the acquisition of task-relevant feature components. Extensive experiments and analysis on three public HSI datasets suggest the superiority of the proposed method. The code will be available on the website at https://github.com/Qba-heu/FDFSL.

The existing methods for hyperspectral image classification (HSIC) mainly focus on the extraction of spectral and spatial features while paying less attention to the interaction of each other. Besides, most of them directly use a parameterized classifier as the final layer of the network. While this design is convenient for end-to-end optimization with the backbone, it overlooks the utilization of the metric space. In this article, a novel hyperspherical structural-aware distillation enhanced spatial–spectral bidirectional interaction network (HSDBIN) is proposed for HSIC. HSDBIN uses a dual-branch design combining the 1-D CNN and transformer to separately learn the detailed spectral correlations and global spatial relationships in parallel. Then, by interacting and aggregating the independent information between two parallel branches, a bidirectional interaction block across branches is designed to explore complementary clues between spectral and spatial pipelines. Finally, to enhance the utilization of metric space and keep compact intraclass relationship, we propose a hyperspherical structural-aware distillation (HSD) to transfer the geometric relationship of hyperspherical space into the metric space of output logits. Extensive experiments and analysis on three public HSI datasets suggest the superiority of the proposed method and verify the effectiveness of the proposed modules.

Deep learning has received much attention in hyperspectral image (HSI) classification. However, most deep learning methods design relatively complex feature extraction and processing network modules for the characteristics of HSIs, which may not be necessary for relatively simple patch-based HSI classification tasks. The complex network structure and high feature channel dimension lead to large computational complexities, which limit the practical applicability of HSI. In this article, an elegant lightweight HSI classification-based information redistribution network (LIRnet) is proposed to separate and reaggregate the feature information to achieve feature information homogenization and extract discriminative feature information, respectively. The classification performance of LIRnet is better than that of existing methods on three different datasets in different scenarios, which proves its effectiveness. In addition, experiments on embedded devices verify the computational efficacy of LIRnet.

Change detection is a prominent research direction in the field of remote sensing image processing. However, most current change detection methods focus solely on detecting changes without being able to differentiate the types of changes, such as “appear” or “disappear” of objects. Accurate detection of change types is of great significance in guiding decision-making processes. To address this issue, this article introduces the object fine-grained change detection (OFCD) task and proposes a method based on frequency decoupling interaction (FDINet). Specifically, in order to enhance the model’s ability to detect change types and improve its robustness to temporal information, a temporal exchange framework is designed. Additionally, to better capture spatial–temporal correlation in bi-temporal features, a wavelet interaction module (WIM) is proposed. This module utilizes wavelet transform for frequency decoupling, separating features into different components based on their frequency magnitudes. Then the module applies different interaction methods according to the characteristics of these frequency components. Finally, to aggregate complementary information from different-scale feature maps and enhance the representational capabilities of the extracted features, a feature aggregation and upsampling module (FAUM) is adopted. A series of experiments show the superiority of FDINet over most state-of-the-art methods, achieving good results on three different datasets.

Feature pyramid network transformer decoder (FPNFormer) module, which can effectively deal with the strong rotation arbitrary of remote sensing images while improving the expressiveness and robustness of the model. It is a plug-and-play module that can be well transferred to various detection models and significantly improves performance. Specifically, we use the computational method of transformer decoder to deal with the problem that the image has any orientation, and its output weakly depends on the order of the input data. We apply it to the feature fusion stage and design two ways top-down and down-top to fuse features of different scales, which enables the model to have a more vital ability to perceive objects at different scales and angles. Experiments on commonly used benchmarks (DOTA1.0, DOTA1.5, SSDD, and RSDD) demonstrate that the proposed FPNFormer module significantly improves the performance of multiple arbitrary-oriented object detectors, such as 1.99% map improvement of rotated retinanet on DOTA’s cross-validation set. On RSDD datasets, the baseline model using FPNFormer improves the map of large objects by 5.1%. Combined with more competitive models, the proposed method can achieve a 79.39% map on the DOTA1.0 dataset. The code is available athttps://github.com/bityangtian/FPNFormer.

Hyperspectral videos (HSVs) offer not only spatial information but also diagnostic spectral features. Due to the fact that spectral features are only related to the material of the object, this advantage can address the issue of RGB video tracking failure when the object and background are visually similar. However, the effectiveness of deep learning models is limited due to insufficient HSV training data. Existing methods tend to divide a hyperspectral image (HSI) into several three-channel false-color images to leverage the existing RGB trackers for transfer learning. Nonetheless, these methods lack adequate exploration of band interrelations and overlook correlation among objects prior to similarity calculation. In this article, a transformer-based band regrouping and feature refinement network (TBR-Net) is introduced, which is specifically tailored for hyperspectral object tracking. To maximize the potential of the RGB tracker and enhance the use of available training data, we propose a transformer-based band regrouping (TBR) method. By modeling long-range spectral dependencies, the inherent context information among bands is captured, which is subsequently utilized to reorganize bands into several false-color images. Furthermore, to combine the relationship of the template and the search (T & S) frames into a correlation calculation, a feature refinement module (FRM) is designed. The cross-attention mechanism enables mutual relation modeling, allowing similar regions to be perceived and form discriminative feature representation. As a result, a hyperspectral tracker can be efficiently trained via transfer learning to address the data insufficiency challenge, while the mutual perception between objects further enhances the tracking performance. Its effectiveness is validated by extensive benchmark experiments, which demonstrate that the TBR-Net surpasses state-of-the-art methods.

The semantic segmentation of wide-field remote sensing images plays a significant role in many fields. However, due to the complexity of the content of remote sensing images, the dataset often has an uneven distribution of land type between different classes and large gaps in the scales of different objects. This often creates great problems for fine segmentation. To solve the issues, an unbalanced class learning network with Scale-adaptive perception (UCSANet) is proposed, which can adaptively cope with Multi-scale objects and unbalanced classes. The design can be inserted in any convolution network easily and can enrich features without increasing too many parameters. The network groups feature and uses atrous convolutions with different dilated rates on different groups to extract Multi-scale features while separable convolutions reduce the amount of network parameters. Then, the fusion of features between different scales is achieved through the self-attention mechanism. Furthermore, a weight map is designed to adaptively combine the predictions of two segmentation heads with Cross-Entropy loss and Lovasz-Softmax loss respectively, which enable the network to focus on learning low-frequency classes without affecting high-frequency classes. Experimental results on GF-6 MSI datasets demonstrate that the proposed UCSANet performs significantly better than others and achieves multi-class segmentation more accurately.

The usability of infrared remote sensing data is often compromised by thin cloud cover. To address this problem, we proposed the multiscale residual fusion network (MRF-Net) to remove thin cloud from infrared remote sensing imagery. Initially, we developed a thin cloud simulation method utilizing Perlin noise and affine transformation to generate high-fidelity thin cloud representations. Subsequently, to accurately discern and eliminate thin cloud from infrared images, we proposed MRF-Net. This model incorporates a multiscale feature fusion module (MSFFM) for extracting shallow features, a residual dense network module (RDNM) for in-depth feature extraction, a residual Swin transformer module (RSTM) for capturing global features, and attention mechanisms to selectively enhance target information. The Swin transformer, a hierarchical Transformer whose representation is computed with shifted windows, is employed to improve the efficiency of global feature extraction. Finally, we devised a combined loss function that accounts for both intrablock and interblock constraints to ensure de-clouding consistency across different image blocks. The intrablock constraint focuses on removing thin cloud within each image block, while the interblock constraint is designed to enhance the consistency of cloud removal between blocks. We have assembled a dataset comprising both simulated and real data to validate the efficiency of our proposed method. Experimental results have shown that our method effectively eliminates thin cloud and surpasses existing state-of-the-art methods. The source codes are available athttps://github.com/CastleChen339/MRF-Net.

In recent years, deep learning-based methods have exhibited remarkable performance in the field of hyperspectral image (HSI) classification. However, conventional supervised methods heavily rely on a substantial number of labeled samples. Self-supervised learning, as a prominent unsupervised representation learning technique, offers the potential to extract valuable information from unlabeled data. In this article, we introduce a novel unsupervised approach called self-supervised learning with the multiscale densely connected network (SS-MSDCNet) to make full use of unlabeled samples for HSI classification. First, a two-stream structure was designed to generate more positive pairs, which enables the contrast self-supervised training to learn more useful information from unlabeled data. Subsequently, a data augmentation technique based on spectral splitting was proposed to coordinate the two-stream structure of SS-MSDCNet, enhancing spectral information expression. The backbone of the proposed approach is the multiscale densely connected network (MSDCNet), which obtains input HSIs with various spatial scales by removing peripheral pixels from the original input and subsequently extracts multiscale spatial-spectral features using 3-D densely connected modules and 3-D spatial attention modules. The 3-D densely connected module effectively harnesses multiscale features extracted by various convolutional layers, while the 3-D spatial attention module enhances the network’s focus on features conducive to accurate classification. To validate the efficacy of our approach, we conducted extensive experiments using four distinct HSI datasets. The results unequivocally demonstrate that SS-MSDCNet outperforms several well-established supervised and unsupervised classification methods. Furthermore, we designed a transfer experiment to confirm SS-MSDCNet’s robust generalization capabilities. The code is available athttps://github.com/mrblank99/SS-MSDCNet.

Training a deep-learning classifier notoriously requires hundreds of labeled samples at least. Many practical hyperspectral image (HSI) scenarios suffer from a substantial cost associated with obtaining a number of labeled samples. Few-shot learning (FSL), which can realize accurate classification with prior knowledge and limited supervisory experience, has demonstrated superior performance in the HSI classification. However, previous few-shot classification algorithms assume that the training and testing data are distributed in the same domains, which is a stringent assumption in realistic applications. To alleviate this limitation, we propose a cross-domain FSL based on graph convolution contrast (GCC-FSL). The proposed method leverages cross-domain learning to acquire transferable knowledge from the source domain for classifying samples in the target domain. Specifically, a positive and negative pairs module is designed for constructing positive and negative pairs by matching the class prototypes of the target domain with those of the source domain, which aligns the data distribution of the source and target domains. In addition, a graph convolution contrast (GCC) module is proposed for extracting global graph-structure information of HSI to improve the ability of feature expression and constructing a graph-contrast loss to solve a domain-shift problem. Finally, a multiscale feature extraction network is designed to expand convolutional receptive fields through feature reuse and increase information interaction for fine-grained feature extraction. The experimental results demonstrate the improved performance for the proposed FSL framework relative to both state-of-the-art convolutional neural network (CNN)-based methods as well as other few-shot techniques. The source code of this method can be found athttps://github.com/JieW-ww/GCC-FSL.

To infer unknown remote sensing scenarios, for remote sensing scene classification (RSSC) most existing deep neural networks (DNNs) are trained on closed datasets. When the acquisition speed and quantity of remote sensing images increases rapidly, these models cannot be used to classify new scenes. Currently, incremental learning as an effective solution for solving thecatastrophic forgettingissue, but ignoringthe stability-plasticity dilemma. In this paper, we propose a new incremental learning network, named efficient channel attention-based multiscale depthwise network (ECA-MSDWNet), in which efficient channel attention (ECA) improve the model’s ability to focus on critical information in complex context and multiscale depthwise convolution (MSDW Conv) extracts multiscale features in a fine-grained way. In addtion, in incremental learning process, we expand new modules based on a dynamic-structure method to fit the residuals between the labels and the outputs of the old model, enhancing the plasticity of the new model for new tasks while maintaining the performance of the old tasks. Finally, we compress the model to reduce redundant parameters and feature dimensions through an effective knowledge distillation strategy. Experiments on four open datasets demonstrate the effectiveness of our method. Our code is available at https://github.com/zhangyu-chd/ECA-MSDWNet.

Change detection (CD) in hyperspectral image (HSI) is of great importance in the remote sensing area. The HSI-CD method based on deep learning (DL) has shown significant progress in achieving precise detection performance. However, many existing methods overlook cross-domain challenges in the CD task. In addition, the scarcity of annotated samples makes the DL models prone to overfitting. To address these issues, a generative cross-domain CD (GCCD) network based on a domain generalization (DG) technique is proposed. GCCD consists of a generator and a discriminator. The generator, with a Morph encoder (ME) and a Semantic encoder (SE), preserves fundamental structural information while introducing randomization to style and content. The discriminator extracts change information for discrimination through dual-temporal images and their difference map. Supervised adversarial learning between the generator and discriminator enhances the model’s ability to extract domain-invariant information. Extensive experiments on various datasets demonstrate the superior performance of the proposed method.

With the development of remote sensing technology, more and more data sources are available for landcover classification tasks, such as hyperspectral images (HSIs), light detection and ranging (LiDAR) data, and synthetic aperture radar (SAR) data. Due to the unique information carried by different sources, the collaborative use of multiple remote sensing data has become a key research direction in landcover classification tasks. Most existing methods are only capable of dealing with two types of remote-sensing images, limiting the potential for the collaboration of more sources. Also, the traditional feature fusion method uses addition or concatenation manner to integrate information, which makes it difficult to make full use of complement characteristics between modalities. As a remedy, we propose a novel multimodal adaptive modulation network (MAMNet), for landcover classification tasks using multimodal remote sensing data. First, the cross-modal interacting module (CIM) is utilized for information absorption between modalities. The feature representation is enhanced, and the modality-specific information is preserved. Second, the modal attention layer (MAL) is designed for multimodal feature fusion. Softmax attention is utilized to eliminate redundant information among the three modal features. Finally, an adaptive multimodal margin loss (AMM loss) is proposed to balance the consistency and diversity of multimodal features. It encourages adjustable decision margins between sources, which enables the model to better utilize complementary information between modalities and partially avoids model-overfitting by defining a more difficult learning target. Experimental results on two benchmark remote sensing datasets show the effectiveness of the proposed method compared with several state-of-the-art approaches.

As an invasive plant in wetlands, Spartina alterniflora (S. alterniflora) causes immeasurable damage to wetland ecosystems. Observing S.alterniflora using multitemporal remote sensing data helps us better understand its further development and facilitates effective containment of its invasion trend. However, inconsistent representation across remote sensing data from different time periods poses a challenge. Fortunately, the utilization of unsupervised domain adaptation (UDA) techniques helps in addressing such issues and enables the exploration of rich temporal dimension information in multitemporal remote sensing data, revealing the spatio-temporal distribution characteristics of S.alterniflora. However, existing UDA methods mostly focus on directly aligning the global or intraclass distribution representations across domains, which overlooks the issue of significant differences between extreme domains and lacks exploration of interclass relationships. To address these limitations, an intermediate domain prototype class-level learning network (IDPNet) is proposed. IDPNet utilizes dynamically generated intermediate domain (ID) features to construct class prototypes while incorporating interclass information into the prototype construction, achieving the class-centered distribution alignment for adaptation. Moreover, intermediate domain feature generation module (IFM) is employed in IDPNet to blend the latent representations from various domains and generate ID features in real time. Additionally, the hierarchical feature fusion module (HFM) is designed to enable IDPNet to learn more discriminative and robust spatio-temporal distribution features, thereby reducing the loss of information from patches. Experimental results on two cross-year multispectral datasets demonstrate that the proposed IDPNet outperforms several state-of-the-art UDA methods.

Advances in multisource remote sensing have allowed for the development of more comprehensive observation. The adoption of deep convolutional neural networks (CNN) naturally includes spatial-spectral information, which has achieved promising performance in multisource data classification. However, challenges are still found with the extraction of spatial distribution and spectrum relationships, which eventually limit the classification performance. To solve the issue, a spatial-spectral perception network (S2PNet) is proposed to extract the advantages of different data sources and the cross information between data sources in a targeted manner. Specifically, the spatial perception network is developed to build the spatial distribution relationship from high-resolution images, while the spectral perception network extracts the spectrum relationship from spectral images. For perceiving cross information, a memory unit is utilized to store the features from different data sources in succession. In addition, the distance loss and reconstruction loss are introduced to keep the feature integrity, and the cross-entropy loss ensures that features can distinguish different classes. The comprehensive experiments are conducted on several datasets to validate the superiority of the proposed algorithm. The proposed S2PNet outperforms the considered classifiers with an average improvement of +0.77%, +5.62%, +1.58%, and +1.79% for overall accuracy values.

To significantly enhance the performance of point cloud semantic segmentation, this manuscript presents a novel method for constructing large-scale networks and offers an effective lightweighting technique. First, a latent point feature processing (LPFP) module is utilized to interconnect base networks such as PointNet++ and Point Transformer. This intermediate module serves both as a feature information transfer and a ground truth supervision function. Furthermore, in order to alleviate the increase in computational costs brought by constructing large-scale networks and better adapt to the demand for terminal deployment, a novel point cloud lightweighting method for semantic segmentation network (PCLN) is proposed to compress the network by transferring multidimensional feature information of large-scale networks. Specifically, at different stages of the large-scale network, the structure and attention information of the point features are selectively transferred to guide the compressed network to train in the direction of the large-scale network. This paper also solves the problem of representing global structure information of large-scale point clouds through feature sampling and aggregation. Extensive experiments on public datasets and real-world data demonstrate that the proposed method can significantly improve the performance of different base networks and outperform the state-of-the-art.

Hyperspectral images (HSIs), with hundreds of narrow spectral bands, are increasingly used for ground object classification in remote sensing. However, many HSI classification models operate pixel-by-pixel, limiting the utilization of spatial information and resulting in increased inference time for the whole image. This paper proposes SegHSI, an effective and efficient end-to-end HSI segmentation model, alongside a novel training strategy. SegHSI adopts a head-free structure with cluster attention modules and spatial-aware feedforward networks (SA-FFN) for multiscale spatial encoding. Cluster attention encodes pixels through constructed clusters within the HSI, while SA-FFN integrates depth-wise convolution to enhance spatial context. Our training strategy utilizes a student-teacher model framework that combines labeled pixel class information with consistency learning on unlabeled pixels. Experiments on three public HSI datasets demonstrate that SegHSI not only surpasses other state-of-the-art models in segmentation accuracy but also achieves inference time at the scale of seconds, even reaching sub-second speeds for full-image classification. Code is available at https://github.com/huanliu233/SegHSI.

Cross-domain methods have been proposed to learn the domain invariant knowledge that can be transferred from the source domain to the target domain. Existing cross-domain methods attempt to minimize the distribution discrepancy of the domains. However, these methods fail to explore the domain invariant subspace due to the samples of different classes between two domains may overlap in the new subspace. They consider the features in the original space data that may be unnecessary or irrelevant to the final classification, and neglect to preserve the local manifold structure between two domains. To solve these problems, a novel feature extraction method called Locality Cross-domain Discriminant Analysis (LCDA) is proposed. LCDA first aligns the distributions and avoids overlap between two domains. Then, LCDA exploits the local manifold structure to maintain the discriminative capability of the low-dimensional projection matrices. Finally, a robust constraint is utilized to preserve the robustness of the projection matrices. The proposed LCDA not only avoids overlap between different classes but also explores the local manifold information. Experiment results on the medical membranous nephropathy hyperspectral dataset demonstrate that the proposed LCDA has better performance than other relevant feature extraction methods.

Oral squamous cell carcinoma (OSCC) has the characteristics of early regional lymph node metastasis. OSCC patients often have poor prognoses and low survival rates due to cervical lymph metastases. Therefore, it is necessary to rely on a reasonable screening method to quickly judge the cervical lymph metastastic condition of OSCC patients and develop appropriate treatment plans. In this study, the widely used pathological sections with hematoxylin-eosin (H&E) staining are taken as the target, and combined with the advantages of hyperspectral imaging technology, a novel diagnostic method for identifying OSCC lymph node metastases is proposed. The method consists of a learning stage and a decision-making stage, focusing on cancer and non-cancer nuclei, gradually completing the lesions' segmentation from coarse to fine, and achieving high accuracy. In the learning stage, the proposed feature distillation-Net (FD-Net) network is developed to segment the cancerous and non-cancerous nuclei. In the decision-making stage, the segmentation results are post-processed, and the lesions are effectively distinguished based on the prior. Experimental results demonstrate that the proposed FD-Net is very competitive in the OSCC hyperspectral medical image segmentation task. The proposed FD-Net method performs best on the seven segmentation evaluation indicators: MIoU, OA, AA, SE, CSI, GDR, and DICE. Among these seven evaluation indicators, the proposed FD-Net method is 1.75%, 1.27%, 0.35%, 1.9%, 0.88%, 4.45%, and 1.98% higher than the DeepLab V3 method, which ranks second in performance, respectively. In addition, the proposed diagnosis method of OSCC lymph node metastasis can effectively assist pathologists in disease screening and reduce the workload of pathologists.

Hyperspectral anomaly detection, which is aimed at distinguishing anomaly pixels from the surroundings in spatial features and spectral characteristics, has attracted considerable attention due to its various applications. In this article, we propose a novel hyperspectral anomaly detection algorithm based on adaptive low-rank transform, in which the input hyperspectral image (HSI) is divided into a background tensor, an anomaly tensor, and a noise tensor. To take full advantage of the spatial–spectral information, the background tensor is represented as the product of a transformed tensor and a low-rank matrix. The low-rank constraint is imposed on frontal slices of the transformed tensor to depict the spatial–spectral correlation of the HSI background. Besides, we initialize a matrix with predefined size and then minimize its$l_{2.1}$-norm to adaptively derive an appropriate low-rank matrix. The anomaly tensor is constrained with the$l_{2.1.1}$-norm to depict the group sparsity of anomalous pixels. We integrate all regularization terms and a fidelity term into a non-convex problem and develop a proximal alternating minimization (PAM) algorithm to solve it. Interestingly, the sequence generated by the PAM algorithm is proven to converge to a critical point. Experimental results conducted on four widely used datasets demonstrate the superiority of the proposed anomaly detector over several state-of-the-art methods.

Fusion with corresponding finer-resolution images has been a promising way to enhance hyperspectral images (HSIs) spatially. Recently, low-rank tensor-based methods have shown advantages compared with other kind of ones. However, these current methods either relent to blind manual selection of latent tensor rank, whereas the prior knowledge about tensor rank is surprisingly limited, or resort to regularization to make the role of low rankness without exploration on the underlying low-dimensional factors, both of which are leaving the computational burden of parameter tuning. To address that, a novel Bayesian sparse learning-based tensor ring (TR) fusion model is proposed, named as FuBay. Through specifying hierarchical sprasity-inducing prior distribution, the proposed method becomes the first fully Bayesian probabilistic tensor framework for hyperspectral fusion. With the relationship between component sparseness and the corresponding hyperprior parameter being well studied, a component pruning part is established to asymptotically approaching true latent rank. Furthermore, a variational inference (VI)-based algorithm is derived to learn the posterior of TR factors, circumventing nonconvex optimization that bothers the most tensor decomposition-based fusion methods. As a Bayesian learning methods, our model is characterized to be parameter tuning-free. Finally, extensive experiments demonstrate its superior performance when compared with state-of-the-art methods.

Deep-learning-based methods are widely used in multisource remote-sensing image classification, and the improvement in their performance confirms the effectiveness of deep learning for classification tasks. However, the inherent underlying problems of deep-learning models still hinder the further improvement of classification accuracy. For example, after multiple rounds of optimization learning, representation bias and classifier bias are accumulated, which prevents the further optimization of network performance. In addition, the imbalance of fusion information among multisource images also leads to insufficient information interaction throughout the fusion process, thus making it difficult to fully utilize the complementary information of multisource data. To address these issues, a Representation-enhanced Status Replay Network (RSRNet) is proposed. First, a dual augmentation including modal augmentation and semantic augmentation is proposed to enhance the transferability and discreteness of feature representation, to reduce the impact of representation bias in the feature extractor. Then, to alleviate the classifier bias and maintain the stability of the decision boundary, a status replay strategy (SRS) is built to regulate the learning and optimization of the classifier. Finally, aiming to improve the interactivity of modal fusion, a novel cross-modal interactive fusion (CMIF) method is employed to jointly optimize the parameters of different branches by combining multisource information. Quantitative and qualitative results on three datasets demonstrate the superiority of RSRNet in multisource remote-sensing image classification, and its outperformance compared with other state-of-the-art methods.

Mixup-based data augmentation has been proven to be beneficial to the regularization of models during training, especially in the remote-sensing field where the training data is scarce. However, in the process of data augmentation, the Mixup-based methods ignore the target proportion in different inputs and keep the linear insertion ratio consistent, which leads to the response of label space even if no effective objects are introduced in the mixed image due to the randomness of the augmentation process. Moreover, although some previous works have attempted to utilize different multimodal interaction strategies, they could not be well extended to various remote-sensing data combinations. To this end, a multistage information complementary fusion network based on flexible-mixup (Flex-MCFNet) is proposed for hyperspectral-X image classification. First, to bridge the gap between the mixed image and the label, a flexible-mixup (FlexMix) data augmentation strategy is designed, where the weight of the label increases with the ratio of the input image to prevent the negative impact on the label space because of the introduction of invalid information. More importantly, to summarize diverse remote-sensing data inputs including various modal supplements and uncertainties, a multistage information complementary fusion network (MCFNet) is developed. After extracting the features of hyperspectral and complementary modalities [X-modal, including multispectral, synthetic aperture radar (SAR), and light detection and ranging (LiDAR)] separately, the information between complementary modalities is fully interacted and enhanced through multiple stages of information complement and fusion, which is used for the final image classification. Extensive experimental results have demonstrated that Flex-MCFNet can not only effectively expand the training data, but also adequately regularize different data combinations to achieve state-of-the-art performance.

Most domain adaptation (DA) methods in cross-scene hyperspectral image classification focus on cases where source data (SD) and target data (TD) with the same classes are obtained by the same sensor. However, the classification performance is significantly reduced when there are new classes in TD. In addition, domain alignment, as one of the main approaches in DA, is carried out based on local spatial information, rarely taking into account nonlocal spatial information (nonlocal relationships) with strong correspondence. A graph information aggregation cross-domain few-shot learning (Gia-CFSL) framework is proposed, intending to make up for the above-mentioned shortcomings by combining FSL with domain alignment based on graph information aggregation. SD with all label samples and TD with a few label samples are implemented for FSL episodic training. Meanwhile, intradomain distribution extraction block (IDE-block) and cross-domain similarity aware block (CSA-block) are designed. The IDE-block is used to characterize and aggregate the intradomain nonlocal relationships and the interdomain feature and distribution similarities are captured in the CSA-block. Furthermore, feature-level and distribution-level cross-domain graph alignments are used to mitigate the impact of domain shift on FSL. Experimental results on three public HSI datasets demonstrate the superiority of the proposed method. The codes will be available from the website: https://github.com/YuxiangZhang-BIT/IEEE_TNNLS_Gia-CFSL.

Domain adaption (DA) is a challenging task that integrates knowledge from source domain (SD) to perform data analysis for target domain. Most of the existing DA approaches only focus on single-source-single-target setting. In contrast, multisource (MS) data collaborative utilization has been extensively used in various applications, while how to integrate DA with MS collaboration still faces great challenges. In this article, we propose a multilevel DA network (MDA-NET) for promoting information collaboration and cross-scene (CS) classification based on hyperspectral image (HSI) and light detection and ranging (LiDAR) data. In this framework, modality-related adapters are built, and then a mutual-aid classifier is used to aggregate all the discriminative information captured from different modalities for boosting CS classification performance. Experimental results on two cross-domain datasets show that the proposed method consistently provides better performance than other state-of-the-art DA approaches.

With the recent development of the joint classification of hyperspectral image (HSI) and light detection and ranging (LiDAR) data, deep learning methods have achieved promising performance owing to their locally sematic feature extracting ability. Nonetheless, the limited receptive field restricted the convolutional neural networks (CNNs) to represent global contextual and sequential attributes, while visual image transformers (VITs) lose local semantic information. Focusing on these issues, we propose a fractional Fourier image transformer (FrIT) as a backbone network to extract both global and local contexts effectively. In the proposed FrIT framework, HSI and LiDAR data are first fused at the pixel level, and both multisource feature and HSI feature extractors are utilized to capture local contexts. Then, a plug-and-play image transformer FrIT is explored for global contextual and sequential feature extraction. Unlike the attention-based representations in classic VIT, FrIT is capable of speeding up the transformer architectures massively and learning valuable contextual information effectively and efficiently. More significantly, to reduce redundancy and loss of information from shallow to deep layers, FrIT is devised to connect contextual features in multiple fractional domains. Five HSI and LiDAR scenes including one newly labeled benchmark are utilized for extensive experiments, showing improvement over both CNNs and VITs.

Recently, many arbitrary-oriented object detection (AOOD) methods have been proposed and applied to remote sensing and other fields. For aerial platforms, lightweight structure and multimodal adaptations of convolutional neural network (CNN) models are urgently needed. Due to the limited model size, the performance of existing lightweight AOOD methods is low, especially in multimodal tasks. In this paper, a multimodal knowledge distillation (MKD) method is proposed for AOOD in aerial images. In MKD, a multimodal dynamic label assignment strategy is designed to select the optimal positive samples dynamically to adapt to different modalities and environments. Different multimodal localization and feature distillation modules are designed to make multimodal knowledge to be complementary and effectively learned by the lightweight model. Experiments on the public dataset demonstrated the effectiveness and advancement of MKD.

The large-scale pre-training image-text foundation models have excelled in a number of downstream applications. The majority of domain generalization techniques, however, have never focused on mining linguistic modal knowledge to enhance model generalization performance. Additionally, text information has been ignored in hyperspectral image classification (HSI) tasks. To address the aforementioned shortcomings, a Multi-modal Domain Generalization Network (MDG) is proposed to learn cross-domain invariant representation from cross-domain shared semantic space. Only the source domain (SD) is used for training in the proposed method, after which the model is directly transferred to the target domain (TD). Visual and linguistic features are extracted using the dual-stream architecture, which consists of an image encoder and a text encoder. A generator is designed to obtain extended domain (ED) samples that are different from SD. Furthermore, linguistic features are used to construct a cross-domain shared semantic space, where visual-linguistic alignment is accomplished by supervised contrastive learning. Extensive experiments on two datasets show that the proposed method outperforms state-of-the-art approaches.

How to use multi-source data for semantic segmentation is a hot topic. In this article, a new multi-source image data semantic segmentation based on multi-source mutual supervision (MMS) has been proposed. First, multi-source data from the same region are trained separately using identical segmentation networks for initialization; then, MMS performs mutual supervision training on these initialized networks, thus adaptively cooperating differences in information distribution between multiple sources of image data, and combines real label constraints output consistency across different networks. Experimental results demonstrate that the proposed MMS strategy can effectively coordinate information between multi-source data, improve segmentation accuracy, and is effective in different semantic segmentation networks and different types of data sets.

Classification of tree species is of great significance to forest surveys. Recently, considering the low differences of spectral information among tree species, enhancing the dependence between long-distance bands has become a research hotspot. A tree species classification method based on a convolutional (2-dimension) long short-term memory (Conv2DLSTM) network and transformer is proposed. First, the main features of HSI are retained by principal component analysis (PCA). Then, the Conv2DLSTM network obtains the global correlation information between long-distance band pixels, and the 3-dimensional convolutional neural network (3DCNN) updates the local spatial-spectral information. Finally, low-level features are converted into semantic tags to guide the modeling of high-level semantic features. The experimental results on the forest dataset demonstrate that the proposed method is superior to other competitive work.

Hyperspectral image (HSI) classification using vision Transformers is an area of active research in the field of remote sensing. However, vision Transformers suffer from lack of inductive bias inherent in convolutional neural networks (CNNs), which hinders their generalization ability when working with limited amounts of data. Furthermore, some research indicates that the competitive performance of vision Transformers primarily stems from the overall architecture, characterized by alternating residual layers of sequence mixers and multi-layer perceptron (MLP) blocks, rather than solely relying on the multi-head self-attention (MSA) mixer module. To this end, a novel CNN-mixer hierarchical spectral Transformer applied to hyperspectral image classification is proposed. To extract spectral features, a hierarchical spectral Transformer is designed to aggregate discriminative features from the spectral dimension. To remedy the lack of inductive bias inherent in vision Transformers, a simple CNN mixer is substituted for the MSA mixer to capture the local spatial features of each HSI window patch. Accordingly, the proposed network architecture leverages advantages of both CNNs and vision Transformers. Through extensive experiments, we find that the proposed method outperforms the state-of-the-art (SOTA) methods applied to the University of Houston, Botswana, and Salinas HSI datasets.

Recently, convolutional neural networks (CNNs) have been developed for remote sensing image fusion (RSIF). To obtain competitive fusion performance, network design becomes more complicated by stacking convolutional layers deeper and wider. However, problems still remain when applying existing networks in practical applications. On the one hand, researchers focus on improving spatial resolution but ignore that the fused images will be used in subsequent interpretation applications, e.g., objection detection. On the other hand, RSIF involves different tasks with different image sources e.g., pansharpening of the panchromatic and multispectral image, hypersharpening of the panchromatic and hyperspectral image, etc. However, existing networks only solve one of them, failing to be compatible with other tasks. To address the above problems, a convenient task-inspired multiscale nonlocal-attention network (MNAN) is proposed for RSIF. The proposed MNAN focuses more on enhancing the multi-scale targets in the scene when improving the resolution of the fused image. In addition, the proposed network can be applied to both pansharpening and hypersharpening tasks without any modification.

With the development of the sensor technology, complementary data of different sources can be easily obtained for various applications. Despite the availability of adequate multisource observation data, for example, hyperspectral image (HSI) and light detection and ranging (LiDAR) data, existing methods may lack effective processing on structural information transmission and physical properties alignment, weakening the complementary ability of multiple sources in the collaborative classification task. The complementary information collaboration manner and the redundancy exclusion operator need to be redesigned for strengthening the semantic relatedness of multisources. As a remedy, we propose a structural optimization transmission framework, namely, structural optimization transmission network (SOT-Net), for collaborative land-cover classification of HSI and LiDAR data. Specifically, the SOT-Net is developed with three key modules: 1) cross-attention module; 2) dual-modes propagation module; and 3) dynamic structure optimization module. Based on above designs, SOT-Net can take full advantage of the reflectance-specific information of HSI and the detailed edge (structure) representations of multisource data. The inferred transmission plan, which integrates a self-alignment regularizer into the classification task, enhances the robustness of the feature extraction and classification process. Experiments show consistent outperformance of SOT-Net over baselines across three benchmark remote sensing datasets, and the results also demonstrate that the proposed framework can yield satisfying classification result even with small-size training samples.

A hyperspectral video contains frames with numerous spectral bands, providing fine reflectance information for object identification and tracking. Enriched features can be learned from spectral-spatial data using deep learning models. However, due to the difficulty in hyperspectral video collection, deep model training is often insufficient, causing reduced performance during the testing stage. To address this issue, we present a novel Band Attention Grouping-based Siamese framework (SiamBAG) for hyperspectral object tracking. SiamBAG employs massive color object tracking data to train a deep neural network. Band weights obtained by band attention module are used to group a hyperspectral image into multiple three-channel false-color images with approximate total group weights. Then multiple enhanced images obtained by histogram equalization are fed to the proposed SiamBAG network to generate a classification branch, a regression branch and a scale tuning branch. In the classification branch, the response maps of multiple groups are fused by regularized group weights to estimate the position of objects. Then the regression branch is used to obtain the initial object position of objects. The position offsets are fed back to the scale tune branch to relocate and fine-tune the object position by exploiting the similarity between template features and detection features. Experimental results demonstrate that the proposed tracker achieves superior tracking performance than other methods. The source codes of this paper will be released at https://github.com/zephyrhours/Hyperspectral-Object-Tracking-SiamBAG.

Currently, the use of rich spectral and spatial information of hyperspectral images (HSIs) to classify ground objects is a research hotspot. However, the classification ability of existing models is significantly affected by its high data dimensionality and massive information redundancy. Therefore, we focus on the elimination of redundant information and the mining of promising features and propose a novel Bole convolution (BC) neural network with a tandem three-direction attention (TDA) mechanism (BTA-Net) for the classification of HSI. A new BC is proposed for the first time in this algorithm, whose core idea is to enhance effective features and eliminate redundant features through feature punishment and reward strategies. Considering that traditional attention mechanisms often assign weights in a one-direction manner, leading to a loss of the relationship between the spectra, a novel three-direction (horizontal, vertical, and spatial directions) attention mechanism is proposed, and an addition strategy and a maximization strategy are used to jointly assign weights to improve the context sensitivity of spatial–spectral features. In addition, we also designed a tandem TDA mechanism module and combined it with a multiscale BC output to improve classification accuracy and stability even when training samples are small and unbalanced. We conducted scene classification experiments on four commonly used hyperspectral datasets to demonstrate the superiority of the proposed model. The proposed algorithm achieves competitive performance on small samples and unbalanced data, according to the results of comparison and ablation experiments. The source code for BTA-Net can be found athttps://github.com/vivitsai/BTA-Net.

To solve the problem of low target detection accuracy caused by the related quantities such as background, target and noise contained in hyperspectral images (HSIs), considering the use of the spatial spectrum and spectral characteristics while increasing the degree of discrimination between target and background, a coarse-to-fine hyperspectral image target detection algorithm based on low-rank tensor decomposition (HTDLTD) is proposed. The HTD based on low rank sparse decomposition mainly decomposes hyperspectral images in spectral dimension, which does not make full use of the spatial information of HSIs, resulting in low detection accuracy. In order to solve this problem, in view of the fact that the hyperspectral third-order tensor can describe the spatial information and spectral information of HSIs equally, the HTD method based on low-rank tensor decomposition (LRTD) is proposed to extract pure background information. Then, in order to solve the problem of low detection accuracy in the case of low target and background discrimination, the rough target detection method based on max over (SMF-MAX) target detection method is proposed to perform rough detection on the original HSI to obtain rough detection results. Finally, in order to further improve the performance of target detection, the fine target detection method based on spectral distance is proposed. By calculating the spectral distance between the original HSI and the synthesized HSI, the final reconstructed target detection result is obtained. Experimental results on three data sets show that the proposed HTDLTD exceeds eight state-of-the-art target detection methods used for comparison.

Infrared cirrus detection is extensively used in military and civil fields, but it poses several challenges to existing methods. These challenges include the complex and diverse shapes of the cirrus clouds, as well as their varying sizes. Additionally, false alarms can be easily triggered by the shadows of cirrus clouds and strong edges. Furthermore, dim and weak cirrus clouds blend into the background and lack distinct features, leading to missed detections. This paper presents an innovative infrared cirrus detection model based on multi-directional graph learning and local fractal feature prior weight mapping to overcome these challenges. Taking into account the structural characteristics of cirrus and background continuity, the proposed method utilizes graph learning enhanced matrix decomposition to separate the cirrus clouds and background. Additionally, to highlight cirrus clouds while suppressing strong edges in the background caused by mountains or rivers, weighted local fractal features are proposed as prior knowledge. To improve the detection of dim and small cirrus clouds as well as accelerate the convergence, a reweighting optimization scheme is proposed. The model is solved using the Alternating Direction Method of Multipliers (ADMM) framework. Extensive experiments demonstrate that the proposed scheme outperforms a variety of classic techniques in terms of detection performance.

Convolutional neural networks (CNN) have attracted increasing attention in the field of multimodal cooperation. Recently, the adoption of CNN-based methods has achieved remarkable performance in multisource remote sensing data classification. However, it is still confronted with challenges in the aspect of complementarity extraction. In this paper, the adversarial complementary learning strategy is embedded into the CNN model called ACL-CNN, which is employed to extract the complementary information of the multisource data. The proposed ACL-CNN is able to filter out the common patterns and specific patterns from multisource data by conducting the adversarial max-min game. Especially, the modality-independent common patterns constitute the basic representation of the land-covers, while the specific patterns that are linearly independent of the common patterns that provide the supplementary representation. Therefore, the complementary information is mapped to a compact and discriminative representation. To eliminate the singularity noise, a learnable pattern sampling module (PSM) is designed to extract the mutual-exclusion relationship between specific patterns. Extensive experiments over three datasets demonstrate the superiority of the proposed ACL-CNN compared with several classification technologies.

Hyperspectral and multispectral images (HS/MS) fusion and classification as an important branch of data quality improvement and interpretation, has attracted increasing attention in recent years. However, the unavailable sensor prior still limits the performance of many traditional fusion methods, consequently deteriorating the classification results. Despite the unsupervised methods based on convolutional neural network (CNN) making a lot of attempts to mitigate the limitations, challenges with extracting the long-range dependencies hamper the performance. To address these impediments, a transformer-based baseline constructed by the cross-scale mixing attention (CSMFormer) is designed for HS/MS fusion and classification. Especially, the spatial-spectral mixer (SSMixer) is utilized to extract the long-range dependencies at large scale. Simultaneously, cross-scale feature calibration is achieved by combining information from the original scale. After that, nonlinear enhancement module (NLEM) is designed to encourage feature discrimination. Note that the spatial and spectral mixers can be replaced by any spatial-spectral feature extractors. Therefore, the proposed CSMFormer is flexible in data fusion, land-covers classification, segmentation, etc. Experiments about data fusion and land-covers classification on two HS/MS wetland remote sensing scenes demonstrate the superiority of the proposed CSMFormer baseline, improving the data quality and classification precision.

Change detection (CD) in synthetic aperture radar (SAR) images aims to detect changed areas by considering the changes in backscattering coefficients. However, the changes can be further divided into positive and negative changes in terms of the increase or decrease of backscattering coefficient, so the CD task can be divided into binary and ternary according to the number of existent categories. This paper introduces an M-nary (binary or ternary) SAR change detection procedure based on the generative adversarial network (GAN) and neural architecture search (NAS) strategy to detect which changes exist in the SAR image-pair and design specialized classifiers for both binary and ternary CD. First, a difference image generation approach based on the salient changed region extraction and neighborhood information is designed for a robust difference representation on the M-nary CD. Due to the further subdivision of changes, the insufficiency of labeled data presents the M-nary change detection with a dilemma. Concerning the lack of labeled information, this paper presents a labeled sample generation strategy based on the GAN architecture search to supplement sample data. Since GAN training is inherently unstable, NAS provides an effective means of searching GAN architecture automatically and ameliorates the reliability of generated samples. During the architecture search procedure, a double-phase evolutionary search strategy is introduced to further improve the stability of GAN training. The experimental results with theoretical analysis prove the validity, robustness, and potential of our method in synthetic as well as real SAR datasets.

Patch-based spatial dictionary has been widely used to fuse a high-resolution panchromatic (PAN) image with a low-resolution multispectral (LMS) image under the framework of sparse representation. However, patch-based dictionary in the spatial domain is not sufficient to preserve spectral information, which may lead to large spectral distortion. To solve this problem, a new spectral dictionary based pansharpening method using subpixel-shift decomposition and joint optimization (termed as PANDA) is proposed. In this method, the model of pansharpening is formulated in a decomposed spectral domain under the sparse and low-rank constraint, as a joint optimization procedure of spectral dictionary and its coefficients. Specifically, a subpixel-shift decomposition is firstly constructed, to decompose the PAN image into a series of subimages with the same spatial resolution of the LMS image. Then, a new imaging model for the pansharpening problem of the LMS image and the decomposed PAN subimages is formulated, with sparse and low-rank constraints. And finally, a joint optimization procedure for the spectral dictionary and its coefficients are theoretically derived, using the spectral information provided by the LMS image and the spatial information provided by the entire PAN subimages, respectively. Experimental results on different datasets show that, the pansharpening performance of the proposed PANDA method outperforms the state-of-the-art methods in both spatial and spectral domains.

Object detection in remote sensing is a challenging task due to the arbitrary orientations of objects and the vast variation in the number of objects within a single image. For instance, one image may contain hundreds of small vehicles, while another may only have a single football field. Recently, DEtection TRansformer (DETR) and its variants have achieved great success in object detection by setting a fixed number of object queries and using bipartite graph matching for one-to-one label assignment. However, we have observed that bipartite graph matching can result in relative redundancy of object queries when the number of objects changes dramatically in an image. This relative redundancy can cause two problems: slower convergence during training and redundant bounding boxes during inference. To analyze the aforementioned problems, we proposed a metric, Redundancy of Object Query (ROQ), to quantitatively analyze the redundancy. Through experiments, we discovered that the reason for the two issues is the difficulty in distinguishing between high-quality negative samples and positive samples. In this paper, we proposed Efficient-Matching Oriented Object Detection with Transformers(EMO2-DETR) consisting of three dedicated components to address the aforementioned issues. Specifically, Reassign Bipartite Graph Matching (RBGM) is proposed to extract high-quality negative samples from the negative samples. And Ignored Sample Predicted Head (ISPH) is proposed to predict high-quality negative samples. Then, Reassigned Hungarian loss is used to better involve high-quality negative samples in the update of model parameters. Extensive experiments on DOTAv1 and DOTAv1.5 datasets demonstrated that our proposed method achieves competitive results.

For the abundant spectral and spatial information recorded in hyperspectral images (HSIs), fully exploring spectral-spatial relationships has attracted widespread attention in hyperspectral image classification (HSIC) community. However, there are still some intractable obstructs. For one thing, in the patch based processing pattern, some spatial neighbor pixels are often inconsistent with the central pixel in land-cover class. For another thing, linear and nonlinear correlations between different spectral bands are vital yet tough for representing and excavating. To overcome these mentioned issues, an adaptive mask sampling and manifold to Euclidean subspace learning (AMS-M2ESL) framework is proposed for HSIC. Specifically, an adaptive mask based intra-patch sampling (AMIPS) module is firstly formulated for intra-patch sampling in an adaptive mask manner based on central spectral vector oriented spatial relationships. Then, based on distance covariance descriptor, a dual channel distance covariance representation (DC-DCR) module is proposed for modeling unified spectral-spatial feature representations and exploring spectral-spatial relationships, especially linear and nonlinear interdependence in spectral domain. Furthermore, considering that distance covariance matrix lies on the symmetric positive definite (SPD) manifold, we implement a manifold to Euclidean subspace learning (M2ESL) module respecting Riemannian geometry of SPD manifold for high-level spectral-spatial feature learning. Additionally, we introduce an approximate matrix square-root (ASQRT) layer for efficient Euclidean subspace projection. Extensive experimental results on three popular HSI data sets with limited training samples demonstrate the superior performance of the proposed method compared with other state-of-the-art methods. The source code is available at https://github.com/lms-07/AMS-M2ESL.

Spectral reconstruction (SR) aims to recover the hyperspectral images (HSIs) from the corresponding RGB images directly. Most SR studies based on supervised learning require massive data annotations to achieve superior reconstruction performance, which are limited by complicated imaging techniques and laborious annotation calibration in practice. Thus, unsupervised strategies attract attention of the community, however, existing unsupervised SR works still face a fatal bottleneck from low accuracy. Besides, traditional CNN-based models are good at capturing local features but experience difficulty in global features. To ameliorate these drawbacks, we propose an unsupervised SR architecture with strong constraints, especially constructing a novel Masked Transformer (MFormer) to excavate latent hyperspectral characteristics to restore realistic HSIs further. Concretely, a Dual Spectral-wise Multi-head Self-attention (DSSA) mechanism embedded in transformer is proposed to firmly associate multi-head and channel dimensions and then capture the spectral representation in the implicit solution spaces. Furthermore, a plug-and-play Mask-guided Band Augment (MBA) module is presented to extract and further enhance the band-wise correlation and continuity to boost the robustness of the model. Innovatively, a customized loss based on the intrinsic mapping from HSIs to RGB images and the inherent spectral structural similarity is designed to restrain spectral distortion. Extensive experimental results on three benchmarks verify that our MFormer achieves superior performance over other state-of-the-art supervised and unsupervised methods under a no-label training process equally.

Although spatio-spectral and spatio-temporal fusion has been well explored, few efforts are made on integrating spatio-spectral-temporal features. As an intrinsic prior, low tensor-rank has been successfully taken into effect by current fusion models, most of which, however, resort to establishing an overall low-rank norm without performing factorization techniques thus have trouble capturing the latent high-order structure of hyperspectral data cube. To address that, a novel Anisotropicly Sparse (AS) tensor norm is developed to make the rank minimization a learnable process under Tucker decomposition. The AS norm enables the model to minimize the multi-linear tensor ranks if imposed on the core tensor after factorization, hence it significantly improves the model’s fusion performance. In the temporal domain, a Hadamard-product based variability descriptor is incorporated into the fusion model to map the former information to current time. Additionally, piece-wise smooth prior of the Tucker factors is employed by extra regularizers as supplement to the loss spatial information. With the Proximal Differential Matrix being developed for optimization, the proposed method reaches state-of-the-art results on both spatio-spectral and spatio-spectral-temporal fusion at low computational cost.

Infrared remote sensing images are becoming increasingly popular due to their superior penetration and resistance to light interference. However, challenges still remain when applying them in real-world applications: 1) raw infrared images suffer from severe stripes interference, and the preprocessing techniques used to obtain standard image products for subsequent detection tasks tend to be time-consuming, which fails to meet the application requirements; 2) current destriping techniques may inevitably weaken the local contrast between some objects and the local background since they need to consider the gray consistency of the overall image; 3) in low-resolution images, dim and small infrared targets are challenging to discriminate, resulting in high false alarms. To address these challenges, we proposed a progressive task-based universal network for raw infrared image ship detection while simultaneously removing stripes. First, we built an integrated network consisting of two components: the stripe denoising component (SDC) and the object detection component (ODC). We also designed a feedback loss adjustment mechanism to enhance the focus of the SDC on the target area. Second, a directed two-branch network was constructed for efficient stripe noise removal, including anx-direction branch for feature enhancement and ay-direction branch for grayscale smoothing. Finally, a parallel network with two labels was designed to extract the inherent features of the target and the background, as well as their relationship features, to achieve refined ship detection. We conducted experiments on a self-assembled dataset from the GaoFen-1 satellite to validate our approach. The experimental results demonstrated that the proposed method outperformed other state-of-the-art methods in infrared image ship detection.

Ship detection and tracking from remote sensing image sequences has become an increasingly important research point. However, there are still many challenges for ship tracking from low-resolution remote sensing image sequences: 1) the dim and small objects contain only a few shape and texture features, making it difficult to detect and track ships; 2) broken clouds often resemble ships, resulting in false tracking; 3) the ship may be occluded by clouds leading to missed tracking. To address these challenges, we proposed a novel multi-level supervision network for ship tracking from low-resolution remote sensing image sequences. First, we designed a gradient difference-guided object clarification network component to significantly improve the object saliency, which is also implemented based on the multi-frame correlation enhancement images to improve the feature strength of small targets in the input data. Second, to reduce the difficulty of completing complex tasks, a multi-level supervised network framework with multiple components was presented to achieve improving the target clarity, detecting targets and tracking targets step-by-step. Finally, to improve the trajectory integrity and tracking accuracy, a joint tracking method based on a low frame rate tracking criterion was proposed to control the state of target tracking module. The method was validated on a self-assembled dataset from the GaoFen-4 satellite. The experiment results show the stronger competitive and accuracy of the proposed method than other state-of-the-art object tracker.

In hyperspectral image (HSI) classification, objects corresponding to pixels of different classes exhibit varying size characteristics, which causes a challenge for effective pixelwise feature extraction and classification. In this article, we propose a novel multiscale model, called multiarea target attention (MATA). The proposed MATA uses an architecture that includes a shared feature extractor (FE) and classifier to capture multiscale spectral–spatial information effectively and efficiently. The FE uses a multiscale target attention module (MSTAM) to extract spectral–spatial information from target pixels and their similar pixels across multiscale areas, while$L_{2}$-normalization is used to address discrepancies between features of different scales. The classifier adopts a classwise decision weighting strategy to account for the varying sizes of different classes and the different contributions of semantic features at each scale to each class. Experimental results on five public HSI datasets demonstrate that the proposed MATA outperforms existing state-of-the-art single- and multiscale models, confirming its effectiveness and efficiency in HSI classification. Code is available athttps://github.com/huanliu233/MATA.

Landslide is one of the most dangerous and frequently occurred natural disasters. The semantic segmentation technique is efficient for wide area landslide identification from high-resolution remote sensing images (HRSIs). However, considerable challenges exist because the effects of sediments, vegetation, and human activities over long periods of time make visually blurred old landslides very challenging to detect based upon HRSIs. Moreover, for terrain features like slopes, aspect and altitude variations cannot be sufficiently extracted from 2-D HRSIs but can be from digital elevation model (DEM) data. Then, a feature-fusion based semantic segmentation network (FFS-Net) is proposed, which can extract texture and shape features from 2-D HRSIs and terrain features from DEM data before fusing these two distinct types of features in a higher feature layer. To segment landslides from background, a multiscale channel attention module is purposely designed to balance the low-level fine information and high-level semantic features. In the decoder, transposed convolution layer replaces original mathematical bilinear interpolation to better restore image resolution via learnable convolutional kernels, and both dropout and batch normalization (BN) are introduced to prevent over-fitting and accelerate the network convergence. Experimental results are presented to validate that the proposed FFS-Net can greatly improve the segmentation accuracy of visually blurred old landslides. Compared to U-Net and DeepLabV3+, FFS-Net can improve the mean intersection over union (mIoU) metric from 0.508 and 0.624 to 0.67, the F1 metric from 0.254 and 0.516 to 0.596, and the pixel accuracy (PA) metric from 0.874 and 0.906 to 0.92, respectively. For the detection of visually distinct landslides, FFS-NET also offers comparable detection performance, and the segmentation is improved for visually distinct landslides with similar color and texture to surroundings.

The geological characteristics of old landslides can provide crucial information for the task of landslide protection. However, detecting old landslides from high-resolution remote sensing images (HRSIs) is of great challenges due to their partially or strongly transformed morphology over a long time and thus the limited difference with their surroundings. Additionally, small-sized datasets can restrict in-depth learning. To address these challenges, this paper proposes a new iterative classification and semantic segmentation network (ICSSN), which can significantly improve both object-level and pixel-level classification performance by iteratively upgrading the feature extraction module shared by the object classification and semantic segmentation networks. To improve the detection performance on small-sized datasets, object-level contrastive learning is employed in the object classification network featuring a siamese network to realize global features extraction, and a sub-object-level contrastive learning method is designed in the semantic segmentation network to efficiently extract salient features from boundaries of landslides. An iterative training strategy is also proposed to fuse features in the semantic space, further improving both the object-level and pixel-level classification performances. The proposed ICSSN is evaluated on a real-world landslide dataset, and experimental results show that it greatly improves both the classification and segmentation accuracy of old landslides. For the semantic segmentation task, compared to the baseline, the F1 score increases from 0.5054 to 0.5448, the mIoU improves from 0.6405 to 0.6610, the landslide IoU grows from 0.3381 to 0.3743, the PA is improved from 0.945 to 0.949, and the object-level detection accuracy of old landslides surges from 0.55 to 0.90. For the object classification task, the F1 score increases from 0.8846 to 0.9230, and the accuracy score is up from 0.8375 to 0.8875.

The injection scheme provides an efficient way for CS- and MRA-based pansharpening approaches. Within this paradigm, the estimation of injection gains is one of the keys to pansharpening outcomes, which has attracted much attention in the community. Most of the existing models are derived from the regression methodology. Hence, the reference is indispensable for the estimation. However, the reference is unavailable in practice, and therefore, the estimation is usually performed at a degraded scale. This article is devoted to the estimation of injection gains without reference. A hybrid-scale (HS) estimation, which involves both the high-resolution and low-resolution data, is proposed, along with three HS models. The proposed method features a context-based and fast implementation with fewer tunable parameters. Experimental results show that the HS models yield more accurate and robust results compared with the typical regression-based models, and they are also competitive with the state-of-the-art approaches.

Deep learning has attracted much attention in hyperspectral image(HSI) classification. However, most deep learning methods ignore the information loss during spatial-spectral feature extraction, which potentially affects the classification performance. In this article, Mask Reconstruction-Based Decoupled Convolution Network(MrDCN) is proposed, which including the decoupled feature extraction module (DFEM) to extract spectral information and spatial information of target HSI patch respectively. The reconstruction modules are designed to maintain the feature extraction ability of DFEM and ensure that discriminative information in high-dimensional features and low-dimensional features is preserved. MrDCN outperforms state-of-the-art methods in classification on three datasets of various scenarios, which indicates its effectiveness, and experiments on embedded devices are executed to affirm the efficiency of MrDCN.

Remote sensing scene understanding is a highly challenging task, and has gradually emerged as a research hotspot in the field of intelligent interpretation of remote sensing data. Recently, the use of convolutional neural networks (CNNs) has been proven to be a fruitful advancement. However, with the emergence of visual transformers (ViTs), the limitations of traditional small convolutional kernels in directly capturing a large receptive field have posed significant challenges to their dominant role. Additionally, the fixed neuron connections between different convolutional layers have weakened the practicality and adaptability of the models. Furthermore, the global average pooling also leads to the loss of effective information in the acquired features. In this work, a Large kernel Sparse ConvNet weighted by Multi-frequency Attention (LSCNet) is proposed. Firstly, unlike traditional convolutional neural networks, it utilizes two parallel rectangular convolutional kernels to approximate a large kernel, achieving comparable or even better results than ViTs-based methods. Secondly, an adaptive sparse optimization strategy is employed to dynamically optimize the fixed neuron connections between different convolutional layers, achieving a favorable connectivity pattern for capturing abstract features more accurately. Lastly, a novel multi-frequency attention (MFA) module is used to replace global average pooling (GAP), so as to preserve more useful information while weighting the recognition features, thereby enhancing the discriminative and learning abilities of the model. In the conducted experiments, LSCNet achieves the best recognition results on three well-known remote sensing aerial datasets when compared to the state-of-the-art methods (including ViTs-based methods).

In recent years, remote sensing scene classification is one of research hotspots and has played an important role in the field of intelligent interpretation of remote sensing data. However, various complex objects and backgrounds form a variety of remote sensing scenes through spatial combination and correlation, which brings great challenges to accurately classify different scenes. Among them, the insufficient feature difference brought about the unbalanced change of background and target between inter-class sample and the feature representation inconsistency caused by the difference of representation among the intra-class samples have become obstacles to effectively distinguish different scene images. To address these issues, a Multi-stage Self-Guided Separation Network (MGSNet) is proposed for remote sensing scene classification. First of all, different from the previous work, it attempts to utilize the background information outside the effective target in the image as a decision aid through a target-background separation strategy to improve the distinguish ability between target similarity-background difference samples. In addition, the diversity of feature concerns among different network branches is expanded through contrastive regularization to improve the separation of target-background information. Additionally, a self-guided network is proposed to find common features between intra-class samples and improve the consistency of feature representation. It combines the texture and morphological features of images to guide feature learning, effectively reducing the impact of intra-class differences. Extensive experimental results on three benchmark demonstrate that MGSNet can achieve better classification performance compared to the state-of-the-art approaches.

Recently, many lightweight neural networks have been deployed on airborne or satellite remote sensing platforms for real-time object detection. To bridge the performance gap between lightweight models and complex models, many knowledge distillation (KD) methods are investigated. However, existing KD methods ignore to transfer effective directional knowledge. Meanwhile, knowledge of different subtasks interfere with each other. To this end, a directional alignment instance knowledge distillation (DAIK) method for improving the performance of the lightweight object detection model is proposed. Specifically, an angle distillation (AD) module is developed to combine the circular smooth label and teacher logits to transfer effective directional knowledge. Angular-distance Aspect-ratio Look-up-table (AAL) is incorporated into label assignment and re-weighting loss to enhance the prediction sensitivity of direction and shape in a discrete manner. Sample alignment distillation (SAD) reduces the spatial misalignment by mimicking the teacher model’s distribution of anchor points. Extensive experiments are performed on several public remote sensing object detection datasets, which demonstrates the effectiveness of the proposed DAIK.

Change detection (CD) in remote sensing images is a critical task that has achieved significant success by deep learning. Current networks often employ pixel-based differencing, proportion, classification-based, or feature concatenation methods to represent changes of interest. However, these methods fail to effectively detect the desired changes, as they are highly sensitive to factors such as atmospheric conditions, lighting variations, and phenological variations, resulting in detection errors. Inspired by the Transformer structure, we adopt a cross-attention mechanism to more robustly extract feature differences between bitemporal images. The motivation of the method is based on the assumption that if there is no change between image pairs, the semantic features from one temporal image can well be represented by the semantic features from another temporal image. Conversely if there is a change, there are significant reconstruction errors. Therefore, a Cross Swin Transformer based Siamese U-shaped network namely CSTSUNet is proposed for remote sensing change detection. CSTSUnet consists of encoder, difference feature extraction, and decoder. The encoder is based on a hierarchical Resnet with the Siamese U-net structure, allowing parallel processing of bitemporal images and extraction of multi-scale features. The difference feature extraction consists of four difference feature extraction modules that compute difference feature at multiple scales. In this module, Cross Swin Transformer is employed in each difference feature extraction module to communicate the information of bitemporal images. The decoder takes in the multi-scale difference features as input, injects details and boundaries iteratively level by level, and makes the change map more and more accurate. We conduct experiments on three public datasets, and the experimental results demonstrate that the proposed CSTSUNet outperforms other state-of-the-art methods in terms of both qualitative and quantitative analyses. Our code is available at https://github.com/l7170/CSTSUNet.git.

Recently, deep learning has become one of the most popular tools for pansharpening, many relevant methods have been investigated and reflected great performance. However, a non-negligible problem is the absence of ground-truth (GT). A common solution is using degraded images as training input and the original images are employed as GT. The learned mapping between low resolution (LR) and high resolution (HR) is simulated, is not real, which may cause spectral distortion or insufficient spatial texture enhancement of fused images. In order to address the drawback, a novel unsupervised attention pansharpening net (UAP-Net) is proposed. The proposed UAP-Net mainly contains two major components: 1) the deep residual network (DRN) and 2) spatial texture attention block (STAB). The DRN aims to extract spectral features and spatial details features from low-resolution multi-spectral (LRMS) and panchromatic (PAN), and to fuse those features to make them more representative. The designed STAB adopts the high-frequency component of corresponding input PAN as the weight to enhance the spatial details of the residual block output features. Moreover, a new loss function including two spatial losses and two spectral losses are established. The losses are calculated in the spatial domain and the frequency domain, respectively. Experiments on Gaofen-2 and Worldview-2 remote sensing data demonstrate that the proposed UAP-Net could fuse PAN and LRMS images effectively without the help of high-resolution multi-spectral (HRMS). The proposed framework is fully general and can be used for many multisource remote sensing image fusion, and achieves optimal performance in terms of both the subjective visual effect and the quantitative evaluation.

The process of annotating hyperspectral image (HSI) data is characterized by its time-consuming and labor-intensive nature. To address this challenge, researchers often employ a meta-learning paradigm known as few-shot learning (FSL), which leverages source domains containing a substantial number of labeled samples to assist in the classification of target domains with limited labeled samples. Many existing FSL methods rely on a conditional domain-adversarial strategy to mitigate the domain shift between source and target domains. However, these methods overlook the fact that the degrees of conditional distribution discrepancies between the two domains can vary significantly across different classes, leading to suboptimal conditional distribution alignment. To address this problem, we propose a framework called Adaptive Domain-Adversarial Few-Shot Learning (ADAFSL). Overall, the proposed ADAFSL employs an adaptive strategy that assigns varying weights to the conditional adversarial losses for different classes based on their respective degrees of discrepancies, thereby achieving global conditional distribution alignment. Specifically, a local alignment score map is constructed by measuring the similarity between labeled and unlabeled samples using both Euclidean and class-covariance metrics. This map is then multiplied with the conditional adversarial loss map, thus allocating more emphasis to the classes exhibiting greater discrepancies between the two domains. Moreover, to enhance cross-domain FSL, we design a multi-scale spectral-spatial feature extraction (MSFE) module, which incorporates cascaded multi-scale dilated convolutions. Experimental results on four public HSI datasets demonstrate that the proposed ADAFSL outperforms other state-of-the-art methods. The source code of this method can be found at https://github.com/JieW-ww/ADAFSL.

High-precision remote sensing image object detection has broad application prospects in military defense, disaster emergency, urban planning, and other fields. However, the arbitrary orientation, dense arrangement, and small size of objects in remote sensing images lead to poor detection accuracy of existing methods. To achieve accurate detection, this paper proposes an arbitrary directional remote sensing object detection method, called FANet, based on feature fusion and angle classification. Initially, the angle prediction branch is introduced, and the circular smooth label method is used to transform the angle regression problem into a classification problem, which solves the difficult problem of abrupt changes in the boundaries of the rotating frame while realizing the object frame rotation. Subsequently, to extract robust remote sensing objects, innovative introduce pure convolutional model as a backbone network, while Conv is replaced by GSConv to reduce the number of parameters in the model along with ensuring detection accuracy. Finally, the strengthen connection feature pyramid network (SC-FPN) is proposed to redesign the lateral connection part for deep and shallow layer feature fusion, and add jump connections between the input and output of the same level feature map to enrich the feature semantic information. In addition, add a variable parameter to the original localization loss function to satisfy the bounding box regression accuracy under different IoU thresholds, and thus obtain more accurate object detection. The comprehensive experimental results on two public datasets for rotated object detection DOTA and HRSC2016 demonstrate the effectiveness of our method.

Hyperspectral image (HSI) consists of abundant spectral and spatial characteristics, which contribute to a more accurate identification of materials and land covers. However, most existing methods of hyperspectral image analysis primarily focus on spectral knowledge or coarse-grained spatial information while neglecting the fine-grained morphological structures. In the classification task of complex objects, spatial morphological differences can help to search for the boundary of fine-grained classes, e.g., forestry tree species. Focusing on subtle traits extraction, a spatial-logical aggregation network (SLA-NET) is proposed with morphological transformation for tree species classification. The morphological operators are effectively embedded with the trainable structuring elements, which contributes to distinctive morphological representations. We evaluate the classification performance of the proposed method on two tree species datasets, and the results demonstrate that the proposed SLA-NET significantly outperforms the other state-of-the-art classifiers.

With the rapid development of the unmanned aerial vehicle (UAV) industry, UAV image object detection technology has become a hotspot. However, due to a large number of dense small objects in UAV images, quickly and effectively detecting objects and achieving accurate classification is still a challenge. With this observation, we propose an efficient object detection network for UAV images based on cross-layer feature aggregation (CFANet). Firstly, we design a novel cross-feature aggregation module (CFA) to aggregate features at different scales on the basis of avoiding semantic gaps, so as to replace common features for feature fusion and achieve accurate detection. This method makes up for the defect that the layer-by-layer feature transfer method only focuses on the features of the previous layer and cannot fully integrate spatial and semantic information. Secondly, a layered associative spatial pyramid pooling module (LASPP) is proposed to capture context information while maintaining the sensitivity of feature maps at different layers to detail information. Thirdly, the alpha-IoU loss function is introduced to accelerate the convergence speed of the model and improve the detection accuracy. Finally, an adaptive overlapping slice (AOS) for high-resolution images is proposed to protect the integrity of the object when slicing. To verify the effectiveness of the proposed method, extensive experiments on challenge datasets for object detection in UAV images VisDrone2021 and UAVDT datasets are carried out. The results show that, compared with the other most advanced detectors, the proposed method can achieve significant performance on the basis of ensuring real-time detection.

Text information including extensive prior knowledge about land cover classes has been ignored in hyperspectral image (HSI) classification tasks. It is necessary to explore the effectiveness of linguistic mode in assisting HSI classification. In addition, the large-scale pretraining image–text foundation models have demonstrated great performance in a variety of downstream applications, including zero-shot transfer. However, most domain generalization methods have never addressed mining linguistic modal knowledge to improve the generalization performance of model. To compensate for the inadequacies listed above, a language-aware domain generalization network (LDGnet) is proposed to learn cross-domain-invariant representation from cross-domain shared prior knowledge. The proposed method only trains on the source domain (SD) and then transfers the model to the target domain (TD). The dual-stream architecture including the image encoder and text encoder is used to extract visual and linguistic features, in which coarse-grained and fine-grained text representations are designed to extract two levels of linguistic features. Furthermore, linguistic features are used as cross-domain shared semantic space, and visual–linguistic alignment is completed by supervised contrastive learning in semantic space. Extensive experiments on three datasets demonstrate the superiority of the proposed method when compared with the state-of-the-art techniques. The codes will be available from the website:https://github.com/YuxiangZhang-BIT/IEEE_TGRS_LDGnet.

The detection of camouflaged targets in the complex background is a hot topic of current research. Existing hyperspectral target detection algorithms do not take advantage of spatial information and rarely use temporal information. It is difficult to obtain the required targets, and the detection performance in hyperspectral sequences with complex background will be low. Therefore, a hyperspectral time-series target detection method based on spectral perception and spatial-temporal tensor decomposition (SPSTT) is proposed. Firstly, a sparse target perception strategy based on spectral matching is proposed. To initially acquire the sparse targets, the matching results are adjusted by using the correlation mean of the prior spectrum, the pixel to be measured and the four-neighborhood pixel spectra. The separation of target and background is enhanced by making full use of local spatial structure information through local topology graph representation of the pixel to be measured. Secondly, in order to obtain a more accurate rank and make full use of temporal continuity and spatial correlation, a spatial-temporal tensor model based on the Gamma norm andL2,1norm is constructed. Furthermore, an excellent alternating direction method of multipliers is proposed to solve this model. Finally, spectral matching is fused with spatial-temporal tensor decomposition in order to reduce false alarms and retain more right targets. A 176-band hyperspectral image sequence (BIT-HSIS-I) dataset is collected for the hyperspectral target detection task. It is found by testing on the collected dataset that the proposed SPSTT has superior performance over the state-of-the-art algorithms.

Scene recognition has attracted rising attentions of many researchers in the remote sensing fields, owing to the rapidly advancing of remote sensing devices in recent years. However, images obtained from various sensors dominate diverse sensor-specific characteristics, which will dramatically weaken the model transferability trained on a source data domain to a different target domain on account of the domain shift issues. To mitigate the domain discrepancy, most existing methods attend to align the cross-domain distributions. While the valuable knowledge of semantic relationships between different scenes is generally overlooked, and the underlying correlation across scenes cannot be fully discovered. For the sake of tackling this challenge, we propose an adaptive remote sensing scene recognition network, which can successfully transfer both the discriminative knowledge and cross-scene relationship from source to target. Specifically, in this paper, we acquire sensor-invariant representations in an adversarial manner and realize fine-grained conditional distribution alignment contrastively. In such a way, the tremendous domain gap can be mitigated to a large extent, and the discriminative and well-matched representations will be derived favorably. In addition, we explicitly construct class-wise relationship distributions belonging to two domains respectively and minimize their divergence to conduct semantic relationship knowledge transfer (SRKT), for the purpose of sufficiently unearthing the intrinsic semantic relative structures that can prompt generality of the model in the target domain. Finally, we conduct multiple experiments on representative multi-domain remote sensing benchmarks, and the extensive experimental results demonstrate the superiority of our proposed approach.

The illegal misuse of non-cooperative UAVs poses huge threats to society and life safety. Infrared imaging is reliable to monitor unmanned aerial vehicles (UAVs) and the anti-UAVs technology via infrared images has attracted more and more attention. In order to provide sufficient time for follow-up, UAVs are acquired at long distances, usually exhibiting the features of weak and small. Furthermore, infrared images are usually with low signal-to-clutter ratio (SCR). These factors make the correct detection of UAVs a challenge. Existing methods do not fully exploit the phenomenon that the UAVs are easily isolated, resulting in unsatisfactory detection results. For alleviating the issue, a novel detection method via isolation Forest (iForest) is proposed. In the proposed method, the multi-direction couple-order derivative properties are firstly analyzed, which enlarges the feature difference between UAVs and background. Then, a global iForest is constructed, which takes full advantage of the phenomenon that UAVs are susceptible to being isolated. As far as we know, this is the first time that iForest is constructed in infrared small targets detection field. Furthermore, a local iForest is created, which further eliminates the residual false alarms of the result of global iForest. Experiments on nine sequences demonstrate the performance of the proposed method, which is capable of detecting various UAVs under diverse background.

With the development of remote sensing technology, high-resolution (HR) remote sensing optical images have gradually become the main source of change detection data. Albeit, the change detection for HR remote sensing images still faces challenges: 1) in complex scenes, a region contains a large amount of semantic information, which makes it difficult to accurately locate the boundaries between different semantics in the feature maps and 2) due to the inability to maintain consistent conditions such as light, weather, and other factors when acquiring bitemporal images, confounding factors such as the style of bitemporal data that are not related to change detection can cause detection difficulties. Therefore, a change detection method based on feature interaction and multitask learning (FMCD) is proposed in this article. To improve the ability to detect changes in complex scenes, FMCD models the context information of features through a multilevel feature interaction module, so as to obtain representative features, and to improve the sensitivity of the model to changes, the interaction between two temporal features is realized through the mix attention block (MAB). In addition, to eliminate the influence of weather and other factors, FMCD adopts a multitask learning strategy, takes domain adaptation as an auxiliary task, and maps the features of bitemporal images to the same space through the feature relationship adaptation module (FRAM) and feature distribution adaptation module (FDAM). Experiments on three datasets show that the proposed method is superior to other state-of-the-art methods.

The semantic segmentation of multi-temporal remote sensing images to construct wetland land surface coverage is the basis for the perception and dynamic modeling of geographic scenes. However, the segmentation of Spartina alterniflora (S.alterniflora) in remote sensing images on wetlands faces the problems such as low level for cooperative interpretation in multi-temporal images and high fragmentation in the distribution of S.alterniflora. To solve the issues, a multiple attention network (MARNet) based on transfer learning is proposed. The method is designed with a plug-and-play attention module to enhance the learning of vegetation features and improve the network’s ability to focus on small areas of S.alterniflora. At the same time, MARNet designs the transfer learning architecture from both inter-domain alignment and intra-domain adaptation perspectives,aligning the statistical distribution by using the maximum mean difference (MMD) between the source and target domains, and entropy minimization within the domain of the target domain to enhance the high confidence prediction of this domain. In addition, since the samples have a serious imbalance problem, redundant cutting and splicing steps are employed for the prediction results to prevent the poor edge prediction of some image blocks. Experimental results on three cross-year RSIs datasets demonstrate that the proposed MARNet performs significantly better than other networks and is able to extract S.alterniflora in wetlands more accurately.

Currently, cross-scene hyperspectral image (HSI) classification has drawn increasing attention. It is necessary to train a model only on source domain (SD) and directly transferring the model to target domain (TD), when TD needs to be processed in real time and cannot be reused for training. Based on the idea of domain generalization, a Single-source Domain Expansion Network (SDEnet) is developed to ensure the reliability and effectiveness of domain extension. The method uses generative adversarial learning to train in SD and test in TD. A generator including semantic encoder and morph encoder is designed to generate the extended domain (ED) based on encoder-randomization-decoder architecture, where spatial randomization and spectral randomization are specifically used to generate variable spatial and spectral information, and the morphological knowledge is implicitly applied as domain invariant information during domain expansion. Furthermore, the supervised contrastive learning is employed in the discriminator to learn class-wise domain invariant representation, which drives intra-class samples of SD and ED. Meanwhile, adversarial training is designed to optimize the generator to drive intra-class samples of SD and ED to be separated. Extensive experiments on two public HSI datasets and one additional multispectral image (MSI) dataset demonstrate the superiority of the proposed method when compared with state-of-the-art techniques. The codes will be available from the website:https://github.com/YuxiangZhang-BIT/IEEE_TIP_SDEnet.

Deep learning (DL) based methods represented by convolutional neural networks (CNNs) are widely used in hyperspectral image classification (HSIC). Some of these methods have strong ability to extract local information, but the extraction of long-range features is slightly inefficient, while others are just the opposite. For example, limited by the receptive fields, CNN is difficult to capture the contextual spectral-spatial features from a long-range spectral-spatial relationship. Besides, the success of DL-based methods is greatly attributed to numerous labeled samples, whose acquisition are time-consuming and cost-consuming. To resolve these problems, a hyperspectral classification framework based on multi-attention Transformer (MAT) and adaptive superpixel segmentation-based active learning (MAT-ASSAL) is proposed, which successfully achieves excellent classification performance, especially under the condition of small-size samples. Firstly, a multi-attention Transformer network is built for HSIC. Specifically, the self-attention module of Transformer is applied to model long-range contextual dependency between spectral-spatial embedding. Moreover, in order to capture local features, an outlook-attention module which can efficiently encode fine-level features and contexts into tokens is utilized to improve the correlation between the center spectral-spatial embedding and its surroundings. Secondly, aiming to train a excellent MAT model through limited labeled samples, a novel active learning (AL) based on superpixel segmentation is proposed to select important samples for MAT. Finally, to better integrate local spatial similarity into active learning, an adaptive superpixel (SP) segmentation algorithm, which can save SPs in uninformative regions and preserve edge details in complex regions, is employed to generate better local spatial constraints for AL. Quantitative and qualitative results indicate that the MAT-ASSAL outperforms seven state-of-the-art methods on three HSI datasets.

Microscopic hyperspectral image (MHSI) has received considerable attention in the medical field. The wealthy spectral information provides potentially powerful identification ability when combining with advanced convolutional neural network (CNN). However, for high-dimensional MHSI, the local connection of CNN makes it difficult to extract the long-range dependencies of spectral bands. Transformer overcomes this problem well because of its self-attention mechanism. Nevertheless, transformer is inferior to CNN in extracting spatial detailed features. Therefore, a classification framework integrating transformer and CNN in parallel, named as Fusion Transformer (FUST), is proposed for MHSI classification tasks. Specifically, the transformer branch is employed to extract the overall semantics and capture the long-range dependencies of spectral bands to highlight the key spectral information. The parallel CNN branch is designed to extract significant multiscale spatial features. Furthermore, the feature fusion module is developed to effectively fuse and process the features extracted by the two branches. Experimental results on three MHSI datasets demonstrate that the proposed FUST achieves superior performance when compared with state-of-the-art methods.

Joint classification using multisource remote sensing data for Earth observation is promising but challenging. Due to the gap of imaging mechanism and imbalanced information between multisource data, integrating the complementary merits for interpretation is still full of difficulties. In this article, a classification method based on asymmetric feature fusion, named asymmetric feature fusion network (AsyFFNet), is proposed. First, the weight-share residual blocks are utilized for feature extraction while keeping separate batch normalization (BN) layers. In the training phase, redundancy of the current channel is self-determined by the scaling factors in BN, which is replaced by another channel when the scaling factor is less than a threshold. To eliminate unnecessary channels and improve the generalization, a sparse constraint is imposed on partial scaling factors. Besides, a feature calibration module is designed to exploit the spatial dependence of multisource features, so that the discrimination capability is enhanced. Experimental results on the three datasets demonstrate that the proposed AsyFFNet significantly outperforms other competitive approaches.

In this article, the intrinsic properties of hyperspectral imagery (HSI) are analyzed, and two principles for spectral-spatial feature extraction of HSI are built, including the foundation of pixel-level HSI classification and the definition of spatial information. Based on the two principles, scaled dot-product central attention (SDPCA) tailored for HSI is designed to extract spectral-spatial information from a central pixel (i.e., a query pixel to be classified) and pixels that are similar to the central pixel on an HSI patch. Then, employed with the HSI-tailored SDPCA module, a central attention network (CAN) is proposed by combining HSI-tailored dense connections of the features of the hidden layers and the spectral information of the query pixel. MiniCAN as a simplified version of CAN is also investigated. Superior classification performance of CAN and miniCAN on three datasets of different scenarios demonstrates their effectiveness and benefits compared with state-of-the-art methods.

Anomaly detection in hyperspectral images (HSIs) has attracted considerable interest in the remote-sensing domain, which aims to identify pixels with different spectral and spatial features from their surroundings. Most of the existing anomaly detection methods convert the 3-D data cube to a 2-D matrix composed of independent spectral vectors, which destroys the intrinsic spatial correlation between the pixels and their surrounding pixels, thus leading to considerable degradation in detection performance. In this article, we develop a tensor-based anomaly detection algorithm that can effectively preserve the spatial–spectral information of the original data. We first separate the 3-D HSI data into a background tensor and an anomaly tensor. Then the tensor nuclear norm based on the tensor singular value decomposition (SVD) is exploited to characterize the global low rank existing in both the spectral and spatial directions of the background tensor. In addition, the total variation (TV) regularization is incorporated due to the piecewise smoothness. For the anomaly component, the$l_{2.1}$norm is exploited to promote the group sparsity of anomalous pixels. In order to improve the ability of the algorithm to distinguish the anomaly from the background, we design a robust background dictionary. We first split the HSI data into local clusters by leveraging their spectral similarity and spatial distance. Then we develop a simple but effective way based on the SVD to select representative pixels as atoms. The constructed background dictionary can effectively represent the background materials and eliminate anomalies. Experimental results obtained using several real hyperspectral datasets demonstrate the superiority of the proposed method compared with some state-of-the-art anomaly detection algorithms.

Due to the limitations of single-source data, joint classification using multisource remote sensing data has received increasing attention. However, existing methods still have certain shortcomings when faced with feature extraction from single-source data and feature fusion between multisource data. In this article, a method based on multiscale interactive information extraction (MIFNet) for hyperspectral and synthetic aperture radar (SAR) image classification is proposed. First, a multiscale interactive information extraction (MIIE) block is designed to extract meaningful multiscale information. Compared with traditional multiscale models, it can not only obtain richer scale information but also reduce the model parameters and lower the network complexity. Furthermore, a global dependence fusion module (GDFM) is developed to fuse features from multisource data, which implements cross attention between multisource data from a global perspective and captures long-range dependence. Extensive experiments on the three datasets demonstrate the superiority of the proposed method and the necessity of each module for accuracy improvement.

Domain adaptation techniques have been widely applied to the problem of cross-scene hyperspectral image (HSI) classification. Most existing methods use convolutional neural networks (CNNs) to extract statistical features from data and often neglect the potential topological structure information between different land cover classes. CNN-based approaches generally only model the local spatial relationships of the samples, which largely limits their ability to capture the nonlocal topological relationship that would better represent the underlying data structure of HSI. In order to make up for the above shortcomings, a Topological structure and Semantic information Transfer network (TSTnet) is developed. The method employs the graph structure to characterize topological relationships and the graph convolutional network (GCN) that is good at processing for cross-scene HSI classification. In the proposed TSTnet, graph optimal transmission (GOT) is used to align topological relationships to assist distribution alignment between the source domain and the target domain based on the maximum mean difference (MMD). Furthermore, subgraphs from the source domain and the target domain are dynamically constructed based on CNN features to take advantage of the discriminative capacity of CNN models that, in turn, improve the robustness of classification. In addition, to better characterize the correlation between distribution alignment and topological relationship alignment, a consistency constraint is enforced to integrate the output of CNN and GCN. Experimental results on three cross-scene HSI datasets demonstrate that the proposed TSTnet performs significantly better than some state-of-the-art domain-adaptive approaches. The codes will be available from the website: https://github.com/YuxiangZhang-BIT/IEEE_TNNLS_TSTnet.

Recently, some lightweight convolutional neural network (CNN) models have been proposed for airborne or spaceborne remote sensing object detection (RSOD) tasks. However, these lightweight detectors suffer from performance degradation due to the compromise of limited computing resources on embedded devices. In order to narrow this performance gap, a direction-adaptive knowledge extraction and distillation (DKED) method is proposed. Specifically, a dynamic directional convolution (DDC) is developed to extract the typical arbitrary-oriented features, and a direction-adaptive knowledge distillation (DKD) strategy is designed for guiding the lightweight model to learn the intrinsic knowledge of the RSOD task from the high-performance model. Experiments on public datasets demonstrate that the proposed method can effectively improve the performance of the lightweight RSOD model without additional inference costs.

Improving the spatial resolution of a hyperspectral image (HSI) is of great significance in the remotely sensed field. By fusing a high-spatial-resolution multispectral image (MSI) with an HSI collected from the same scene, hyperspectral and multispectral (HS–MS) fusion has been an emerging technique to address the issue. Extracting complex spatial information from MSIs while maintaining abundant spectral information of HSIs is essential to generate the fused high-spatial-resolution HSI (HS2I). A common way is to learn low-rank/sparse representations from HSI and MSI, then reconstruct the fused HS2I based on tensor/matrix decomposition or unmixing paradigms, which ignore the intrinsic geometry proximity inherited by the low-rank property of the fused HS2I. This study proposes to estimate the high-resolution HS2I via low-rank tensor approximation with geometry proximity as side information learned from MSI and HSI by defined graph signals, which we name GLRTA. Row graph ${\mathcal{G}_r}$ and column graph ${\mathcal{G}_c}$ are defined on the horizontal slice and lateral slice of MSI tensor $\mathcal{M}$ respectively, while spectral band graph ${\mathcal{G}_b}$ is defined on a frontal slice of HSI tensor $\mathcal{H}$. Experimental results demonstrate that the proposed GLRTA can effectively improve the reconstruction results compared to other competitive works.

Most domain adaptation (DA) methods focus on the case where the source data (SD) and target data (TD) with the same classes are obtained by the same sensor in cross-scene hyperspectral image (HSI) classification tasks. However, the classification performance is significantly reduced when there are new classes in TD. In addition, domain alignment is carried out based on local spatial information in most methods, rarely taking into account the non-local spatial information (non-local relationships) with strong correspondence. A Dual Graph Cross-domain Few-shot Learning (DG-CFSL) framework is proposed, trying to make up for the above shortcomings by combining Few-shot Learning (FSL) with domain alignment. Both SD with all label samples and TD with a few label samples are implemented for FSL episodic training. Meanwhile, Intra-domain Distribution Extraction block (IDE-block) is designed to characterize and aggregate the intra-domain non-local relationships. Furthermore, feature- and distribution-level cross-domain graph alignments are used to mitigate the impact of domain shift on FSL. Experimental results on two public HSI data sets demonstrate the effectiveness of the proposed method.

To solve the limitation of labeled samples in hyperspectral image (HSI) classification, cross-scene learning methods are developed recently. However, the disparity caused by environmental variation between HSI scenes is still a challenge. As a supplement, light detection and ranging (LiDAR) data provides elevation and spatial information regardless the variations. In this paper, we propose a multisource cross-scene classification method using fractional fusion and spatial-spectral domain adaptation to reduce disparity between scenes. The spatial information of HSI is preserved by fractional differential masks (FrDM) firstly. Then the LiDAR data is utilized for spectral alignment of HSI. The utilization of LiDAR data reduces the pixel-level disparity between scenes. At last, a spatial-spectral domain adaptation network is proposed for feature extraction and classification. Experimental results on HSI and LiDAR scenes show 5% improvements in overall accuracy compared with state-of-the-art methods.

Focusing on joint classification of Hyperspectral image (HSI) and Light detection and ranging (LiDAR) data, a fractional Fourier image transformer (FrIT) is proposed as a backbone network in this paper. In the proposed FrIT, HSI and LiDAR data are firstly fused at pixel-level. Both multi-source and HSI feature extractors are utilized to capture local contexts. Then, a plug-and-play image transformer FrIT is explored for global contexts and sequential feature extraction. Unlike the attention-based representations in classic visual image transformer (VIT), FrIT is capable of speeding up the transformer architectures massively. To reduce the information loss from shallow to deep layers, FrIT is devised to connect contextual features in multiple fractional domains. At last, to evaluate the performance of FrIT, a new HSI and LiDAR benchmark is provided for extensive experiments, on which the proposed FrIT gains an improvement of 3% over state-of-the-art methods.

Spectral quality is one of the most critical issues that has to be considered in real hyperspectral image (HSI) application. Denoising, destriping, inpainting, deblurring and super-resolution are common techniques to improve the quality of HSIs from different aspects. These techniques have attracted much attention that a diversity of methods, algorithms, tools have been well developed to facilitate the development of HSI restoration. Although effectively improving the quality of HSIs, these technologies mainly focus on recovering an HSI captured in the normal sunlight. It is acknowledged that HSIs are captured via passive imaging mechanisms covering the spectral bands from visible& near-infrared to shortwave infrared spectral range (i.e., around 400nm to 2500nm). The imaging condition limits HSI spectrometers to capture HSIs without sunlight (e.g., in dark environments or night time). In this work, a low-light HSI restoration method is proposed, where we borrow idea of intrinsic decomposition based on Retinex theory in natural image low-light enhancement. Additionally, camera response function that describe the spectral degradation of RGB image and relationship between irradiance and pixel values are employed, respectively. The experimental results validate the effectiveness of the proposed method.

Infrared small target detection (ISTD) has attracted widespread attention and been applied in various fields. Due to the small size of infrared targets and the noise interference from complex backgrounds, the performance of ISTD using convolutional neural networks (CNNs) is restricted. Moreover, the constriant that long-distance dependent features can not be encoded by the vanilla CNNs also impairs the robustness of capturing targets' shapes and locations in complex scenarios. To this end, a multi-patch attention network (MPANet) based on the axial-attention encoder and the multi-scale patch branch (MSPB) structure is proposed. Specially, an axial-attention-improved encoder architecture is designed to highlight the effective features of small targets and suppress background noises. Furthermore, the developed MSPB structure fuses the coarse-grained and fine-grained features from different semantic scales. Extensive experiments on the SIRST dataset show the superiority performance and effectiveness of the proposed MPANet compared to the state-of-the-art methods.

Multi-source joint classification has been extensively investigated in single scenario setting; however, for cross scene (CS) classification, few studies have been conducted for evaluating the collaborative performance of multi-sources. In this paper, using hyperspectral image (HSI) and light detection and ranging (LiDAR) data, we propose a multi-source CS classification method, and build source-related alignment to reduce statistical shift. Both geometrical and statistical alignments are considered to learn common-subspaces of each source with preserving discrimination information. Finally, the aligned features from both sources are integrated for final classification. Experimental results demonstrate the superior of the proposed method over other state-of-the-art CS approaches.

Low-earth-orbit (LEO) satellite mega-constellations promise broadband, low-latency network infrastructure from space for terrestrial users in remote areas. However, they face new QoS bottlenecks from infrastructure mobility due to the fast-moving LEO satellites and earth’s rotations. Both cause frequent space-ground link churns and challenge the network latency, bandwidth, and availability at the global scale. Today’s LEO networks mask infrastructure mobility with fixed anchors (ground stations) but cause single-point bandwidth/latency bottlenecks. Instead, we design LBP to remove the LEO network’s QoS bottlenecks from infrastructure mobility. LBP removes remote terrestrial fixed anchors via geographic addressing for shorter latencies and more bandwidth. It adopts local, orbit direction-aware geographic routing to avoid global routing updates for high network availability. LBP further shortens the routing paths by refining handover policies by satellites’ orbital directions. Our experiments in controlled testbeds and trace-driven emulations validate LBP’s 1.64× network latency reduction, 9.66× more bandwidth, and improve network availability to 100%.

Accurate remote sensing monitoring of wetland ground objects is of great significance for ecological protection. In this letter, a convolutional neural network based on feature intersecting learning (FIL-CNN) is designed for wetland classification using multisource remote sensing data. The multi-layer shift feature fusion (MSFF) and attention feature selection (AFS) modules are designed to extract the complementary merits. Specifically, the MSFF is applied to each feature extraction unit, and the asymmetric information fusion is achieved through the spatial position shift of grouped features. Thus, the diversified feature representation is achieved. In the prediction stage, the AFS is executed to explore the channel mutually exclusive relationship between multisource features, resulting in emphasizing the meaningful features and eliminating the unnecessary ones. The experimental results prove the effectiveness and generalization of the proposed FIL-CNN on the wetland datasets.

Hyperspectral image (HSI) has hundreds of continuous bands that contain a lot of redundant information. Besides, a spatial patch of a hyperspectral cube often contains some pixels different from the center pixel category, which are usually called interference pixels. The existence of such interference pixels has a negative effect on extracting more discriminative information. Therefore, in this letter, a multiattention fusion network (MAFN) for HSI classification is proposed. Compared with the current state-of-the-art methods, MAFN uses band attention module (BAM) and spatial attention module (SAM), respectively, to alleviate the influence of redundant bands and interfering pixels. In this way, MAFN realizes feature reuse and obtains complementary information from different levels by combining multiattention and multilevel fusion mechanisms, which can extract more representative features. Experiments were conducted on two public HSI data sets to demonstrate the effectiveness of MAFN. Our source code is available athttps://github.com/Li-ZK/MAFN-2021.

The effective monitoring of low-altitude slow small (LSS) targets represented by unmanned aerial vehicle (UAV) is a great challenge in the field of security in recent years. Most of the existing infrared (IR) small target algorithms focus on high-altitude target detection. However, the low-altitude background is complex and changeable, and high-intensity suspected targets exist widely. Existing methods usually cause high false alarm or failure detection for LSS targets. In this letter, we propose a novel spatiotemporal saliency method for LSS IR targets in image sequences. First, spatial variance saliency mapping and temporal gray saliency mapping are calculated in spatial domain and temporal domain, respectively. Then, the fusion saliency map is obtained by fusing the spatial saliency map and temporal saliency map. Finally, the target is extracted by a simple adaptive threshold segmentation. The proposed method is verified in five low-altitude IR image sequences. Experimental results demonstrate that the proposed method can achieve better detection performance than the existing state-of-the-art methods for LSS targets.

With the gradual opening of airspace, tracking of noncooperative low-altitude slow-speed small size (LSS) targets is important for the maintenance of security. It is still a challenging problem, especially for complex scenarios and real-time constraints. In this letter, an efficient tracking by relocalization (TRL) framework is proposed for small flying object tracking, aiming to alleviate the issue of losing moving targets in a complex background. Our designed relocalization module consists of a feature-aggregated module and a global search module. On the one hand, a feature-aggregated module is integrated into the designed framework to increase the ability to locate small targets. On the other hand, a global search module is developed to update the tracking performance, which attempts to address missed targets in long-term small object tracking tasks. What needs to be declared is that the basic tracking module cooperates with the relocalization module we designed to achieve the tracking of small targets. Performance evaluation of two small-flying target data sets and comparison with several state-of-the-art approaches demonstrate the effectiveness of the proposed framework.

Infrared images have a wide range of applications in military and civilian fields, including night vision, surveillance, and robotics. However, the most commonly used infrared images are low-resolution (LR), which lack texture details, and existing infrared image super-resolution (SR) algorithms are limited by the lack of spatial information utilization. To solve the above problems, a spatial attention residual network (SAResNet) is proposed. Specifically, the network consists of spatial attention residual block (SARB) with several short skip connections (SSCs). The SARB contains 20 spatial attention blocks (SAB), which adaptively adjusts weights of different spatial regions by considering interdependence between spatial features. Meanwhile, the visible images are considered as complementary sources; thus, a visible-assisted training strategy is designed for the infrared SR process, promoting details preservation. Furthermore, the spatial attention (SA) mechanism is utilized, which focuses more on spatial characteristics of the image and refines the main objects and target boundaries. Experimentally, the proposed method, SAResNet, is compared with existing SR methods, and the effectiveness of the proposed method is demonstrated based on both quantity and quality analyses.

Nowadays, due to the ability of extracting deep features, the deep learning-based anomaly detection (AD) methods for hyperspectral images (HSIs) have been widely studied. However, all these AD methods treat the tasks of feature extraction and AD separately. Besides, most of them also do not make use of abundant spatial information of HSIs. Thus, a spectral–spatial hyperspectral AD method via collaborative representation constraint stacked autoencoders (SSCRSAE) is proposed. First, the collaborative representation constraint is imposed on the stacked autoencoders to extract deep nonlinear features that are more suitable for the collaborative representation-based detector (CRD). Then, CRD is used to for obtaining the preliminary detection result, which is more convenient for real HSIs because of no need for assuming the distribution of the background. Finally, aiming at further improving the SSCRSAE detector’s performance, a novel spectral–spatial AD procedure is designed for calculating the final detection result by considering the spatial information of an HSI. Experimental results express that the proposed SSCRSAE exceeds eight state-of-the-art anomaly detectors used for comparison.

Cross-domain hyperspectral image classification is one of the major challenges in remote sensing, especially for target domain data without labels. Recently, deep learning approaches have demonstrated effectiveness in domain adaptation. However, most of them leverage unlabeled target data only from a statistical perspective but neglect the analysis at the instance level. For better statistical alignment, existing approaches employ the entire unevaluated target data in an unsupervised manner, which may introduce noise and limit the discriminability of the neural networks. In this article, we propose confident learning-based domain adaptation (CLDA) to address the problem from a new perspective of data manipulation. To this end, a novel framework is presented to combine domain adaptation with confident learning (CL), where the former reduces the interdomain discrepancy and generates pseudo-labels for the target instances, from which the latter selects high-confidence target samples. Specifically, the confident learning part evaluates the confidence of each pseudo-labeled target sample based on the assigned labels and the predicted probabilities. Then, high-confidence target samples are selected as training data to increase the discriminative capacity of the neural networks. In addition, the domain adaptation part and the confident learning part are trained alternately to progressively increase the proportion of high-confidence labels in the target domain, thus further improving the accuracy of classification. Experimental results on four datasets demonstrate that the proposed CLDA method outperforms the state-of-the-art domain adaptation approaches. Our source code is available athttps://github.com/Li-ZK/CLDA-2022.

Multi-source image registration is challenging due to intensity, rotation, and scale differences among the images. Considering the characteristics and differences of multi-source remote sensing images, a feature-based registration algorithm named Multi-scale Histogram of Local Main Orientation (MS-HLMO) is proposed. Harris corner detection is first adopted to generate feature points. The HLMO feature of each Harris feature point is extracted on a Partial Main Orientation Map (PMOM) with a Generalized Gradient Location and Orientation Histogram-like (GGLOH) feature descriptor, which provides high intensity, rotation, and scale invariance. The feature points are matched through a multi-scale matching strategy. Comprehensive experiments on 17 multi-source remote sensing scenes demonstrate that the proposed MS-HLMO and its simplified version MS-HLMO+outperform other competitive registration algorithms in terms of effectiveness and generalization.

The monitoring of coastal wetlands is of great importance to the protection of marine and terrestrial ecosystems. However, due to the complex environment, severe vegetation mixture, and difficulty of access, it is impossible to accurately classify coastal wetlands and identify their species with traditional classifiers. Despite the integration of multisource remote sensing data for performance enhancement, there are still challenges with acquiring and exploiting the complementary merits from multisource data. In this article, the depthwise feature interaction network (DFINet) is proposed for wetland classification. A depthwise cross attention module is designed to extract self-correlation and cross correlation from multisource feature pairs. In this way, meaningful complementary information is emphasized for classification. DFINet is optimized by coordinating consistency loss, discrimination loss, and classification loss. Accordingly, DFINet reaches the standard solution-space under the regularity of loss functions, while the spatial consistency and feature discrimination are preserved. Comprehensive experimental results on two hyperspectral and multispectral wetland datasets demonstrate that the proposed DFINet outperforms other competitive methods in terms of overall accuracy.

Semantic segmentation of the remote sensing images (RSIs) has attracted increasing interest in recent years. However, large-area segmentation of the woodland presents challenges. The wide distribution and diverse tree species of the woodland make feature extraction difficult. For this reason, an infrared attention network (InfAttNet) is proposed to extract woodland from multispectral RSIs. InfAttNet has an extra infrared spectral encoder which makes use of the sensitivity of vegetation to near infrared and red edge spectrums. This extra encoder applies learning about vegetation to improve woodland segmentation. Several attention blocks are designed to enhance learning about vegetation features and so improve the performance. In addition, a new dataset is built, containing a large number of woodland RSIs and covering several typical woodland distribution regions in China. The experimental results demonstrate that, compared with other networks, InfAttNet has the highest accuracy and is capable of rapid extraction of the woodland in RSIs.

With the increasing availability of multitemporal hyperspectral imagery, hyperspectral change detection under heterogeneous backgrounds is a challenging task. Due to the complexity of background features, traditional change detection algorithms in the spectral domain cannot effectively detect changed features. A novel method using multiple morphological profiles (MMPs) is proposed for hyperspectral change detection to make full use of spatial information. In the designed framework, first, the max-tree/min-tree strategy is applied to extract different attributes of multitemporal hyperspectral images (HSIs), i.e., area attribute and height attribute. Second, a spectral angle weighted-based local absolute distance (SALA) method is designed to reconstruct the discriminative spectral domain. Then, the absolute distance (AD) is adopted to extract changes in constructed feature domain. Finally, a change map is obtained by guided filtering. Experiments conducted on four real hyperspectral datasets demonstrate that the proposed detector achieves better detection performance.

Hyperspectral video camera captures spatial, spectral and temporal information of moving objects. Traditional object tracking methods developed for color videos have been applied to hyperspectral videos after compressing hundreds of spectral bands into three, which does not fully utilize the wealth spectral information. In order to address this issue, we present a tensor sparse correlation filter with a spatial-spectral weighted regularizer for object tracking. First, tensor processing is employed to reduce the spectral differences of homogeneous background, thereby producing robust spectral structure features. Second, a spatial-spectral weighted regularizer is designed in the correlation filter framework to penalize filter template by suppressing spectral features dissimilar to the center pixel in tracking. Third, a sparse constraint term and tracking context information are incorporated to suppress unexpected peaks in the response map. Finally, a reformulated stacked HOG feature extractor and a two-dimensional adaptive scale search strategy are developed to further improve the tracker’s feature discrimination and scale adaptation capability. Experimental results demonstrate that the proposed method achieves superior tracking performance than traditional correlation filter-based trackers.

Object detection (OD) is an important task of computer vision and has been widely used in many fields, including remote sensing (RS). However, the complex scenes, large-scale variation, and dense instances of RS bring huge challenges to OD. To meet these challenges, a novel Nonlocal-aware Pyramid and Multiscale Multitask Refinement Detector (NPMMR-Det) is proposed. Specifically, nonlocal-aware pyramid attention (NP-Attention) is designed for guiding a neural network model to focus more on efficient features and suppress background noise. Then a multiscale refinement feature pyramid network (MSR-FPN) is proposed to fuse the multiscale context features extracted by the NP-Attention guided neural network and adjust the optimal receptive field. In order to use these features more effectively, a multitask refinement head called MTR-Head, with offset sharing and a modulation mechanism, is developed to refine the feature misalignment between the localization task and the classification task. Extensive experiments performed on two public RS data sets demonstrate that the proposed NPMMR-Det achieves competitive performance compared with state-of-the-art methods.

A few lightweight convolutional neural network (CNN) models have been recently designed for remote sensing object detection (RSOD). However, most of them simply replace vanilla convolutions with stacked separable convolutions (SConvs), which may not be efficient due to a lot of precision losses and may not be able to detect oriented bounding boxes (OBBs). Also, the existing OBB detection methods are difficult to constrain the shape of objects predicted by CNNs accurately. In this article, we propose an effective lightweight oriented object detector (LO-Det). Specifically, a channel separation-aggregation (CSA) structure is designed to simplify the complexity of SConvs, and a dynamic receptive field (DRF) mechanism is developed to maintain high accuracy by customizing the convolution kernel and its perception range dynamically when reducing the network complexity. The CSA-DRF component optimizes efficiency while maintaining high accuracy. Then, a diagonal support constraint head (DSC-Head) component is designed to detect OBBs and constrain their shapes more accurately and stably. Extensive experiments on public data sets demonstrate that the proposed LO-Det can run very fast even on embedded devices with the competitive accuracy of detecting oriented objects.

One of the challenges in hyperspectral image (HSI) classification is that there are limited labeled samples to train a classifier for very high-dimensional data. In practical applications, we often encounter an HSI domain (called target domain) with very few labeled data, while another HSI domain (called source domain) may have enough labeled data. Classes between the two domains may not be the same. This article attempts to use source class data to help classify the target classes, including the same and new unseen classes. To address this classification paradigm, a meta-learning paradigm for few-shot learning (FSL) is usually adopted. However, existing FSL methods do not account for domain shift between source and target domain. To solve the FSL problem under domain shift, a novel deep cross-domain few-shot learning (DCFSL) method is proposed. For the first time, DCFSL tackles FSL and domain adaptation issues in a unified framework. Specifically, a conditional adversarial domain adaptation strategy is utilized to overcome domain shift, which can achieve domain distribution alignment. In addition, FSL is executed in source and target classes at the same time, which can not only discover transferable knowledge in the source classes but also learn a discriminative embedding model to the target classes. Experiments conducted on four public HSI data sets demonstrate that DCFSL outperforms the existing FSL methods and deep learning methods for HSI classification. Our source code is available athttps://github.com/Li-ZK/DCFSL-2021.

Due to rich spectral and spatial information, the combination of hyperspectral and multispectral images (MSIs) has been widely used for Earth observation, such as wetland classification. However, mining of meaningful features and effective fusion of multisource remote sensing data are still urgent problems to be solved. In this article, graph-feature-enhanced selective assignment network (GSANet) is proposed. On the one hand, a graph feature extraction module (GFEM) is designed to extract topological structure information and combine with the rich spectral–spatial information. In particular, the features obtained by convolution are first mapped to the graph feature space, and the graph convolution operation is used to achieve propagation between nodes for preserving topological structure information. Moreover, to reduce the difference of graph features resulting from the mapping function and better explore the complementary properties of multisource data, a novel graph fusion strategy-graph dependence fusion is designed. A transition graph is generated to enhance the association and interaction between different graph features, so as to avoid the information loss caused by simple fusion operation. On the other hand, a selective feature assignment module (SFAM) is developed to adaptively assign weights to different discriminative features. SFAM assigns weights to different features to selectively emphasize informative features and suppress less useful ones. Extensive experiments are conducted on two multisource remote sensing datasets, and the improvement of at least 1.27% and 0.98% compared to other state-of-the-art work demonstrates the superiority of the proposed GSANet.

Low-rank-tensor-approximation (LRTA)-based hyperspectral image (HSI) restoration has drawn increasing attention. However, most of the methods construct a hidden low-rank tensor by utilizing the non-local self-similarity (NLSS) and global spectral correlation (GSC) inherited by HSIs. Although achieving state-of-the-art (SOTA) restoration performance, NLSS and GSC have limitations. NLSS is introduced from natural image denoising to remove spatially independent identically distributed (i.i.d.) Gaussian and impulse noise. While GSC, which is naturally possessed by HSIs, is adopted to maintain the spectral integrity and remove spectrally, i.i.d., degradations. Therefore, NLSS and GSC may not be successfully used for complex HSI restoration tasks, such as destriping, cloud removal and recovery of atmospheric absorption bands. To solve the issue, borrowing the idea from manifold learning, the geometry information characterized by proximity relationship, is integrated with the LRTA to solve the above issue, named as multi-graph-based LRTA (MGLRTA). Different with most of the existing methods, the proposed MGLRTA directly models an HSI as a low-rank tensor and efficiently explores the extra proximity information on the defined graphs that are not only inherited by the low-rank constraints but also naturally possessed in HSIs. A well-posed iterative algorithm is designed to solve the restoration problem. Experimental results on different datasets that cover several severe degradation scenarios demonstrate that the proposed MGLRTA outperforms the SOTA HSI restoration methods.

Detecting small infrared (IR) targets against low-altitude complex background is always a challenge for IR search and tracking (IRST) system due to limited small target characteristics, the moving background caused by camera motion, and extremely cluttered backgrounds. The existing methods usually cause high false alarm or do not work against the chaotic low-altitude complex background. In this article, a novel spatial–temporal tensor model with saliency filter regularization (STTM-SFR) is developed to detect small IR targets. First, the small target detection task is transformed into a sparse and low-rank tensor optimization problem using the spatial–temporal prior knowledge of background and target. The construction of the holistic STTM can retain the complete spatial–temporal information of the original IR image sequence. Then, the SFR term limited between background and foreground aims to promote target saliency learning. That is to say, the SFR term can avoid the offset approximation of the low-rank tensor, so as to recover a clean target image from the original IR tensor. Finally, an effective alternating direction method of multipliers (ADMM) algorithm framework is designed to solve the proposed STTM-SFR model. The effectiveness and robustness of the STTM-SFR model are verified in six real IR scenes. Experimental results show that our method outperforms other baseline methods. Moreover, the proposed STTM-SFR method is more robust than the existing state-of-the-art STTMs against low-altitude moving backgrounds.

Infrared (IR) small target detection in the complex background is an important but challenging research hotspot in the field of target detection. The existing methods usually cause high false alarms in the complex background and fail to make full use of the complete information of the image. In this article, a novel IR small target detection model that combines facet derivative-based multidirectional edge awareness with spatial–temporal tensor (FDMDEA-STT) is presented. First, we construct an STT model (STTM) to transform the target detection problem into a low-rank and sparse tensor optimization problem based on the prior information of the target and background in the spatial–temporal domain. Then, based on the facet derivative, we define a multidirectional edge awareness mapping and fuse it into the STTM as sparse prior information. Finally, an effective algorithm based on the alternating direction method of multipliers (ADMM) is designed to solve the above model. The effectiveness of the proposed method is verified on eight real IR image sequences. Experimental results demonstrate that the proposed method has better detection performance than the existing state-of-the-art methods.

This article proposes a multiscale spectral features graph fusion (MSFGF) method for selecting proper hyperspectral bands. The MSFGF regards that the selected bands should reflect diagnostic spectral information of ground objects at different scales, and it explores band selection from the aspect of multiple spatial scales. First, it adopts the multiscale low-rank decomposition (MSLRD) model to find multiscale spectral features of different ground objects. The model considers divergent spatial structures or spatial correlations of ground objects at different scales, and factorizes the hyperspectral data cube into a series of low-rank block-wise data cubes, where the blocks take spatial structures of different ground objects at increasing scales. Second, the MSFGF presents the multiscale sparse spectral clustering (MSSC) model to fuse the separate connected graphs of multiscale spectral features into a consensus graph. The consensus graph combines the complementary information of multiscale spectral features and helps to reveal the intrinsic clustering structure of all spectral bands. Finally, the MSFGF utilizes spectral clustering to find clusters from the consensus graph and selects representative bands. Experimental results on three widely used hyperspectral data prove the superiority of MSFGF in selecting bands, where it outperforms other seven state-of-the-art methods in classification with an acceptable computational cost.

Arbitrary-oriented object detection (AOOD) plays a significant role in image understanding in remote-sensing scenarios. The existing AOOD methods face the challenges of ambiguity and high costs in angle representation. To this end, a multigrained angle representation (MGAR) method, consisting of coarse-grained angle classification (CAC) and fine-grained angle regression (FAR), is proposed. Specifically, the designed CAC avoids the ambiguity of angle prediction by discrete angular encoding (DAE) and reduces complexity by coarsening the granularity of DAE. Based on CAC, FAR is developed to refine the angle prediction with much lower costs than narrowing the granularity of DAE. Furthermore, an Intersection over Union (IoU)-aware FAR-Loss (IFL) is designed to improve the accuracy of angle prediction using an adaptive reweighting mechanism guided by IoU. Extensive experiments are performed on several public remote-sensing datasets, which demonstrate the effectiveness of the proposed MGAR. Moreover, experiments on embedded devices demonstrate that the proposed MGAR is also friendly for lightweight deployments.

Extracting roads from high-resolution remote sensing images (HRSIs) is vital in a wide variety of applications, such as autonomous driving, path planning, and road navigation. Due to the long and thin shape as well as the shades induced by vegetation and buildings, small-sized roads are more difficult to discern. In order to improve the reliability and accuracy of small-sized road extraction when roads of multiple sizes coexist in an HRSI, an enhanced deep neural network model termed Dual-Decoder-U-Net (DDU-Net) is proposed in this paper. Motivated by the U-Net model, a small decoder is added to form a dual-decoder structure for more detailed features. In addition, we introduce the dilated convolution attention module (DCAM) between the encoder and decoders to increase the receptive field as well as to distill multi-scale features through cascading dilated convolution and global average pooling. The convolutional block attention module (CBAM) is also embedded in the parallel dilated convolution and pooling branches to capture more attention-aware features. Extensive experiments are conducted on the Massachusetts Roads dataset with experimental results showing that the proposed model outperforms the state-of-the-art DenseUNet, DeepLabv3+ and D-LinkNet by 6.5%, 3.3%, and 2.1% in the mean Intersection over Union (mIoU), and by 4%, 4.8%, and 3.1% in the F1 score, respectively. Both ablation and heatmap analysis are presented to validate the effectiveness of the proposed model. Moreover, the designed small decoder and introduced DCAM can be used as a portable module to be embedded in other U-Net-like models with encoder-decoder structure to enhance the road detection performance, especially for small-sized roads. The high portability of the designed module is validated by embedding in the LinkNet, which greatly improves the road segmentation performance.

Hyperspectral image (HSI) classification has drawn increasing attention recently. However, it suffers from noisy labels that may occur during field surveys due to a lack of prior information or human mistakes. To address this issue, this article proposes a novel dual-channel residual network (DCRN) to resolve HSI classification with noisy labels. Currently, the influence of noisy labels is reduced by simply detecting and removing those anomalous samples. Different from such a specifically designed noise cleansing method, DCRN is easy to implement but highly effective. It enhances its model robustness to noisy labels to a great extent by employing a novel dual-channel structure and a noise-robust loss function. In this way, DCRN can mitigate influence from noisy labels while fully utilizing useful information from mislabeled samples for augmented training. Experiments are conducted on several hyperspectral data sets with manually generated noisy labels to demonstrate its excellent performance. The code is available athttps://github.com/Li-ZK/DCRN-2021.

Low-resolution ship detection from optical satellite image sequences is critical in high-orbit remote sensing satellite applications. However, it is still a difficult problem due to the following challenges: 1) the size of the ship is tiny in the low-resolution image; 2) the ship target is dim and the contrast with the background is low; 3) the interference of cloud and fog covering is complex and changeable. For these reasons, the targets are easily lost during the detection. In fact, the Clearer the Objects against to the background, the more Confident the Observers can detect it. In light of these considerations, we propose a COCO-Net to detect the small dynamic objects on low-resolution images in this paper. First, the multi-frame images are associated by introducing motion information as an effective compensation for small object features. Second, an integrated dual-supervised network that processes single-level tasks hierarchically is presented to adaptively enhance the input data quality of object detection without being limited by diverse scene disturbances. Third, a unified ROI-loss scheme that modulates the loss function of the first component by introducing ROI-masks from the second component is utilized to make the first component also work for object detection. In addition, we construct a new dataset for the small dynamic object detection based on the GaoFen-4 satellite imagery. Comprehensive experiments on a self-assembled dataset from the GaoFen-4 satellite show the superior performance of the proposed method compared to state-of-the-art object detectors.

Joint use of multisensor information has attracted considerable attention in the remote sensing community. While applications in land-cover observation benefit from information diversity, multisensor integration technique is confronted with many challenges, including inconsistent size of data, different data structures, uncorrelated physical properties, and scarcity of training data. In this article, an information fusion network, named interleaving perception convolutional neural network (IP-CNN), is proposed for integrating heterogeneous information and improving joint classification performance of hyperspectral image (HSI) and light detection and ranging (LiDAR) data. Specifically, a bidirectional autoencoder is designed to reconstruct hyperspectral and LiDAR data together, and the reconstruction process is trained with no dependence upon annotated information. Both HSI-perception constraint and LiDAR-perception constraint are imposed on multisource structural information integration. Accordingly, fused data are fed into a two-branch CNN for final classification. To validate the effectiveness of the model, the experiments were conducted using three datasets (i.e., Muufl Gulfport data, Trento data, and Houston data). The final results demonstrate that the proposed framework can significantly outperform state-of-the-art methods even with small-size training samples.

Existing infrared small-target detection methods tend to perform unsatisfactorily when encountering complex scenes, mainly due to the following: 1) the infrared image itself has a low signal-to-noise ratio (SNR) and insufficient detailed/texture knowledge; 2) spatial and structural information is not fully excavated. To avoid these difficulties, an effective method based on three-order tensor creation and Tucker decomposition (TCTD) is proposed, which detects targets with various brightness, spatial sizes, and intensities. In the proposed TCTD, multiple morphological profiles, i.e., diverse attributes and different shapes of trees, are designed to create three-order tensors, which can exploit more spatial and structural information to make up for lacking detailed/texture knowledge. Then, Tucker decomposition is employed, which is capable of estimating and eliminating the major principal components (i.e., most of the background) from three dimensions. Thus, targets can be preserved on the remaining minor principal components. Image contrast is further enhanced by fusing the detection maps of multiple morphological profiles and several groups with discontinuous pruning values. Extensive experiments validated on two synthetic data and six real data sets demonstrate the effectiveness and robustness of the proposed TCTD.

Hyperspectral target detection in complex backgrounds is a challenging and important research topic in the remote sensing field. Traditional target detectors consider the background spectrum to obey a Gaussian distribution. However, this distribution may not meet the requirements in real hyperspectral images. In addition, the background and spatial information of most existing target detection algorithms are rarely fully utilized. Therefore, a new weighted Cauchy distance graph (WCDG) and local adaptive collaborative representation detection (CGCRD) is proposed. First, a WCDG similarity measure is designed. In order to adjust the effect of target pixels on the graph model, a weighted Cauchy distance Laplace matrix is constructed, and then the matrix is applied to the matched filter detector. Second, local adaptive collaborative representation strategy is developed. The penalty coefficient is weighted by the local spatial Euclidean distance combined with the Pearson correlation coefficient, and then the detection result is obtained based on the residual. Finally, aforementioned two strategies are fused to fully utilize the spatial and spectral information. A 176-band hyperspectral image (BIT-HSI-I) dataset is collected for the target detection task. The related algorithms are performed on the BIT-HSI-I dataset, and the detection results demonstrate that the proposed algorithm has better detection performance than other state-of-the-art algorithms.

Remote sensing using multisensor platforms has been systematically applied for monitoring and optimizing human activities. Several advanced techniques have been developed to enhance and extract the spatially and spectrally semantic information in the hyperspectral image (HSI) and light detection and ranging (LiDAR) data processing and analysis. However, an abundance of redundant information and sometimes a lack of discriminative features reduce the efficiency and effectiveness of multisource classification methods. This article proposes a fractional Gabor convolutional network (FGCN), focusing on efficient feature fusion and comprehensive feature extraction. First, the proposed FGCN uses Octave convolution layers to perform multisource information fusion and preserve discriminative information. Second, fractional Gabor convolutional (FGC) layers are proposed to extract multiscale, multidirectional, and semantic change features. The completeness and discrimination of the multisource features using different FGC kernels are improved, which yield robust feature extraction against semantic changes. Finally, the fractional Gabor feature and spectral feature are combined with two weighting factors which can be learned during the network training. Experimental results and comparisons with state-of-the-art multisource classification methods indicate the effectiveness of the proposed FGCN. With the FGCN, we can obtain an 89.90% overall accuracy on the challenging Muufl Gulfport (MUUFL) data set, with an improvement of 3% over state-of-the-art methods.

Recently, many arbitrary-oriented object detection (AOOD) methods have been proposed and attracted widespread attention in many fields. However, most of them are based on anchor-boxes or standard Gaussian heatmaps. Such label assignment strategy may not only fail to reflect the shape and direction characteristics of arbitrary-oriented objects, but also have high parameter-tuning efforts. In this paper, a novel AOOD method called General Gaussian Heatmap Label Assignment (GGHL) is proposed. Specifically, an anchor-free object-adaptation label assignment (OLA) strategy is presented to define the positive candidates based on two-dimensional (2D) oriented Gaussian heatmaps, which reflect the shape and direction features of arbitrary-oriented objects. Based on OLA, an oriented-bounding-box (OBB) representation component (ORC) is developed to indicate OBBs and adjust the Gaussian center prior weights to fit the characteristics of different objects adaptively through neural network learning. Moreover, a joint-optimization loss (JOL) with area normalization and dynamic confidence weighting is designed to refine the misalign optimal results of different subtasks. Extensive experiments on public datasets demonstrate that the proposed GGHL improves the AOOD performance with low parameter-tuning and time costs. Furthermore, it is generally applicable to most AOOD methods to improve their performance including lightweight models on embedded platforms.

The problem of effectively exploiting the information multiple data sources has become a relevant but challenging research topic in remote sensing. In this article, we propose a new approach to exploit the complementarity of two data sources: hyperspectral images (HSIs) and light detection and ranging (LiDAR) data. Specifically, we develop a new dual-channel spatial, spectral and multiscale attention convolutional long short-term memory neural network (called dual-channel$A^{3}$CLNN) for feature extraction and classification of multisource remote sensing data. Spatial, spectral, and multiscale attention mechanisms are first designed for HSI and LiDAR data in order to learn spectral- and spatial-enhanced feature representations and to represent multiscale information for different classes. In the designed fusion network, a novel composite attention learning mechanism (combined with a three-level fusion strategy) is used to fully integrate the features in these two data sources. Finally, inspired by the idea of transfer learning, a novel stepwise training strategy is designed to yield a final classification result. Our experimental results, conducted on several multisource remote sensing data sets, demonstrate that the newly proposed dual-channel$A^{\,3}$CLNN exhibits better feature representation ability (leading to more competitive classification performance) than other state-of-the-art methods.

The key to hyperspectral anomaly detection is to effectively distinguish anomalies from the background, especially in the case that background is complex and anomalies are weak. Hyperspectral imagery (HSI) as an image–spectrum merging cube data can be intrinsically represented as a third-order tensor that integrates spectral information and spatial information. In this article, a prior-based tensor approximation (PTA) is proposed for hyperspectral anomaly detection, in which HSI is decomposed into a background tensor and an anomaly tensor. In the background tensor, a low-rank prior is incorporated into spectral dimension by truncated nuclear norm regularization, and a piecewise-smooth prior on spatial dimension can be embedded by a linear total variation-norm regularization. For anomaly tensor, it is unfolded along spectral dimension coupled with spatial group sparse prior that can be represented by the${l}_{2,1}$-norm regularization. In the designed method, all the priors are integrated into a unified convex framework, and the anomalies can be finally determined by the anomaly tensor. Experimental results validated on several real hyperspectral data sets demonstrate that the proposed algorithm outperforms some state-of-the-art anomaly detection methods.

In this article, anomaly detection is considered for hyperspectral imagery in the Gaussian background with an unknown covariance matrix. The anomaly to be detected occupies multiple pixels with an unknown pattern. Two adaptive detectors are proposed based on the generalized likelihood ratio test design procedure and ad hoc modification of it. Surprisingly, it turns out that the two proposed detectors are equivalent. Analytical expressions are derived for the probability of false alarm of the proposed detector, which exhibits a constant false alarm rate against the noise covariance matrix. Numerical examples using simulated data reveal how some system parameters (e.g., the background data size and pixel number) affect the performance of the proposed detector. Experiments are conducted on five real hyperspectral data sets, demonstrating that the proposed detector achieves better detection performance than its counterparts.

Wetland classification is of great significance for monitoring. Recently, collaborative analysis of multisource data has received special attention considering the limitations of single source data. In this paper, a wetland classification method based on feature exchange is proposed. Firstly, the weighting shared residual blocks are utilized for feature extraction. Then, the scaling factors in batch normalization (BN) self-determine the redundancy of current channel, which is replaced by another channel when the scaling factor is less than the threshold. To eliminate unnecessary channels and improve the generalization, sparsity constraint is employed on partial scaling factors. Experimental results on multisource wetland dataset demonstrate that the proposed method outperforms other competitive works.

This paper aims at providing visible and infrared images registered in geometric space for image fusion. Focusing on the characteristics and differences of visible and infrared images, a feature-based registration algorithm is implemented. The key technologies include image scale-space for implementing multi -scale properties, Harris corner detection for keypoints extraction, and partial intensity invariant feature descriptor (PIIFD) for keypoints description. Eventually, a multi-scale Harris- Piifd image registration algorithm framework is proposed. The experimental results of four sets of representative real data show that the algorithm has excellent, stable performance in visible and infrared image registration, and can achieve accurate spatial alignment, which has strong practical application value and certain generalization ability.

Gaofen-6 (GF-6) is a geostationary, earth-observation satellite, rely on it's multi-spectral images, GF-6 has the ability to support the monitoring of woodland resources. In this paper, the multi-spectral images sent by GF-6 are studied as dataset, and a model called Infrared Attention Network (InfAttNet) which based on semantic segmentation method is proposed to distinguish woodland from other land types to achieve the purpose of woodland extraction. To make full use of the spectral information, InfAttNet has an additional encoder to extract the features of infrared bands independently. Besides, infrared attention blocks help InfAttNet to enhance the characteristics of woodland. The experimental results proved that InfAttNet improves the accuracy of woodland extraction, and the segmentation effect is strengthened compared with classical networks.

With the increasing of hyperspectral datasets, multi-temporal hyperspectral change detection has gradually attracted re-searcher's attention. Most of traditional change detection methods only consider spectral information, but ignore importance of spatial structure information, which leads to low detection accuracy. In this work, a novel patch tensor-based change detection method (PTCD) is proposed for hyperspectral imagery to make full use of spatial structure information. Firstly, the tensor decomposition and reconstruction strategies are used to eliminate influence of various factors in multi-temporal dataset. Meanwhile, patch-based strategy is adopted to incorporate the non-overlapping local similar property into the proposed method to exploit spatial structural information. Finally, a specially designed detector is adopted to further improve the detection accuracy. Experiments conducted on two real hyperspectral datasets demonstrate that the proposed detector achieves better detection performance.

Hyperspectral images have high spectral resolution, but this is often at the expense of spatial resolution. Although deep learning-based super-resolution (SR) algorithms have shown comparative performance for spatial resolution enhancement, most of them cannot effectively extract features of different size objects because of single scale convolution. In deep architectures, low level features also tend to disappear during transmission. In this paper, an efficient network (MRBMFF) for enhancing the spatial resolution of hyperspectral image is proposed. Based on the multiscale residual block (MRB), features at different scales can be effectively extracted and fused. Meanwhile, the multilevel feature fusion (MFF) is introduced to concatenate the low and high level features. Effective SR images could be recovered after inputting their low-resolution counterparts to the proposed network. Experimental results show that the proposed network achieves superior reconstruction performance compared with the state-of-the-art approaches.

Cross-scene hyperspectral image (HSI) classification has gradually received widespread attention, because most models perform unsatisfactory classification performance on training and testing samples from two different scenes. At present, the domain adaptation technique is used to solve this problem, most of which only design models from the level of data statistical features, while ignore the potential topological relationships between the land cover classes. In order to make up for the above shortcoming, a domain adaptation based on graph and statistical features is proposed in the papaer. This method uses convolutional neural network (CNN) extracting features with rich semantic information to dynamically construct graphs, and further introduces graph optimal transport (GOT) to align topological relations to assist distribution alignment based on maximum mean discrepancy (MMD). The experimental results on two cross-scene HSI datasets demonstrate the effectiveness of the proposed method.

Robust infrared small-target detection has always been a research hotspot in target search and tracking systems. However, the image itself has low signal-to-noise ratio (SNR), and the targets usually lack detailed/texture information. In addition, the background is complex and diverse. All the above factors make it easy for the targets to be submerged. In this paper, a novel method is proposed based on a three-order creation and the Tucker decomposition. First, the morphological profiles (i.e., area attribute and height attribute of max-tree) are applied to create a three-order tensor, which compensates for the lack of detailed information by supplementing spatial information in infrared images. Then, the Tucker decomposition is employed on the created tensor, in which most of the background can be estimated and eliminated from three dimensions. Finally, the target is detected on the remaining and the results of diverse morphological profiles are fused, which further enhances the target information. Experimental results demonstrate the effectiveness of the proposed method.

Anomaly detection from remote sensing images is to detect pixels whose spectral signatures are different from their background. Anomalies are often man-made targets. With such target signatures being unknown, anomaly detection has many important applications, such as water quality monitoring, crop stress surveying, and law enforcement-related uses, where prior information of targets is often unavailable. The key to success is accurate background modeling. Anomaly detection from remote sensing images is challenging because spatial coverage is very large and the background is highly heterogeneous. For pixel-based anomaly detection, computing cost in background modeling and a spatial-convolution-type detection process is very expensive. Thus, parallel and distributed computing is critical in reducing execution time, which can fit the need for real-time or near real-time detection from airborne and spaceborne platforms in support of immediate decision-making. This article reviews the recent advances in anomaly detection from hyperspectral remote sensing images and their implementation using parallel and distributed systems. The classical methods, i.e., the Reed-Xiaoli (RX) algorithm and its variants, including its real-time processing version, are illustrated in commodity graphic processing units (GPUs), cloud, and field-programmable gate array (FPGA) implementations. Practical issues and future development trends are also discussed.

Recently, the low-rank and sparse decomposition model (LSDM) has been used for anomaly detection in hyperspectral imagery. The traditional LSDM assumes that the sparse component where anomalies and noise reside can be modeled by a single distribution which often potentially confuses weak anomalies and noise. Actually, a single distribution cannot accurately describe different noise characteristics. In this article, a combination of a mixture noise model with low-rank background may more accurately characterize complex distribution. A modified LSDM, by modeling the sparse component as a mixture of Gaussian (MoG), is employed for hyperspectral anomaly detection. In the proposed framework, the variational Bayes (VB) algorithm is applied to infer a posterior MoG model. Once the noise model is determined, anomalies can be easily separated from the noise components. Furthermore, a simple but effective detector based on the Manhattan distance is incorporated for anomaly detection under complex distribution. The experimental results demonstrate that the proposed algorithm outperforms the classic Reed-Xiaoli (RX), and the state-of-the-art detectors, such as robust principal component analysis (RPCA) with RX.

Random Gaussian noise and striping artifacts are common phenomena in hyperspectral images (HSI). In this article, an effective restoration method is proposed to simultaneously remove Gaussian noise and stripes by merging a denoising and a destriping submodel. A denoising submodel performs a multiband denoising, i.e., Gaussian noise removal, considering Gaussian noise variations between different bands, to restore the striped HSI from the corrupted image, in which the striped HSI is constrained by a weighted nuclear norm. For the destriping submodel, we propose an adaptive anisotropy total variation method to adaptively smoothen the striped HSI, and we apply, for the first time, the truncated nuclear norm to constrain the rank of the stripes to 1. After merging the above two submodels, an ultimate image restoration model is obtained for both denoising and destriping. To solve the obtained optimization problem, the alternating direction method of multipliers (ADMM) is carefully schemed to perform an alternative and mutually constrained execution of denoising and destriping. Experiments on both synthetic and real data demonstrate the effectiveness and superiority of the proposed approach.

It is observed that the great spectral variation in the same hyperspectral image (HSI) pixel class often leads to misclassification. To solve this problem, we have proposed an automatic clustering-based two-branch convolutional neural network (CNN): first, to reduce the intraclass spectral variation, the HSI pixels are automatically subdivided into smaller classes by clustering; second, in order to suppress the interference of spectral amplitude variation, the SincNet is introduced to capture the spectral pattern by giving more weight to the spectral shape; third, the DS-CNN with double directional strip convolution kernel is designed to extract spatial feature, so that specific contextual interactional features can be collected, especially in strip-shaped field-like roads and farmlands; finally, the spectral and spatial features extracted by the two branches are fused at fully connected layer to obtain an accurate classification. Extensive experiments demonstrated that the proposed method can obtain better classification performance than the state-of-the-art methods.

Tensor-based fusion that couples the high spatial resolution of a multispectral image (MSI) to the high spectral resolution of a hyperspectral image (HSI) is considered. The fusion problem is first formulated mathematically as a convex optimization of a tensor trace norm imposing low-rank spatially as well as spectrally, with an alternating-directions optimization featuring linearization providing the solution. Although prior tensor-based fusion approaches typically resort to tensor decomposition, the proposed algorithm exploits ideas from the field of tensor completion to directly impose a low-rank property spatially and spectrally while avoiding the computationally complex patch clustering and dictionary learning common to competing fusion techniques. Additionally, small modifications to the basic optimization permit a fusion process robust to missing hyperspectral values such as those that can result from dead stripes in real hyperspectral sensors. The experimental evaluations on both synthetic imagery as well as real imagery demonstrate that the resulting low-rank tensor-approximation (LRTA) fusion algorithm preserves both spatial details and texture, yielding significantly improved image quality when compared to other state-of-the-art fusion methods as well as effective restoration under conditions of missing stripes within the HSI.

Hyperspectral image (HSI) classification is one of the most active research topics and has achieved promising results boosted by the recent development of deep learning. However, most state-of-the-art approaches tend to perform poorly when the training and testing images are on different domains, e.g., the source domain and target domain, respectively, due to the spectral variability caused by different acquisition conditions. Transfer learning-based methods address this problem by pretraining in the source domain and fine-tuning on the target domain. Nonetheless, a considerable amount of data on the target domain has to be labeled and nonnegligible computational resources are required to retrain the whole network. In this article, we propose a new transfer learning scheme to bridge the gap between the source and target domains by projecting the HSI data from the source and target domains into a shared abundance space based on their own physical characteristics. In this way, the domain discrepancy would be largely reduced such that the model trained on the source domain could be applied to the target domain without extra efforts for data labeling or network retraining. The proposed method is referred to as physically constrained transfer learning through shared abundance space (PCTL-SAS). Extensive experimental results demonstrate the superiority of the proposed method as compared to the state of the art. The success of this endeavor would largely facilitate the deployment of HSI classification for real-world sensing scenarios.

Cross-scene classification is one of the major challenges for hyperspectral image (HSI) classification, especially for target scenes without label samples. Most traditional domain adaptive methods learn a domain invariant subspace to reduce statistical shift while ignoring the fact that there may not exist a shared subspace when marginal distributions of source and target domains are very different. In addition, it is important for HSI classification to preserve discriminant information in the original space. To solve this issue, discriminative cooperative alignment (DCA) of subspace and distribution is proposed to cooperatively reduce the geometric and statistical shift. In the proposed framework, both geometrical and statistical alignments are considered to learn subspaces of the two domains with preserving discrimination information. Furthermore, a reconstruction constraint is imposed to enhance the robustness of subspace projection. Experimental results on three cross-scene HSI data sets demonstrate that the proposed DCA is significantly better than some state-of-the-art domain-adaptive approaches.

Infrared small-target detection under heterogeneous background, as a challenging task, plays an important role in many applications. In practice, there are not only bright targets but also dim targets, e.g., rescue aircraft and vehicles in the forest fire scene. Considering that most existing infrared small-target detection methods are merely aimed at bright targets, a novel method using multiple morphological profiles (MMP) is proposed, which can detect various types of targets whose brightness varies greatly. In the designed morphological feature extraction, different attributes, i.e., area attribute and height attribute, are applied to extract spatial size and contrast information of small-target in the max-tree and min-tree, respectively. Furthermore, discontinuous pruning values are further utilized for different attributes, and a designed fusion strategy of different pruning values results in more robust detection performance. Experimental results validated on two synthetic data and six real data sets demonstrate that the proposed MMP can not only detect a variety of brightness of targets and different types of targets and kinds of spatial sizes of targets but also further improve the contrast between targets and background, and the background clutter is significantly suppressed.

Medical hyperspectral imagery has recentlyattracted considerable attention. However, for identification tasks, the high dimensionality of hyperspectral images usually leads to poor performance. Thus, dimensionality reduction (DR) is crucial in hyperspectral image analysis. Motivated by exploiting the underlying structure information of medical hyperspectral images and enhancing the discriminant ability of features, a discriminant tensor-based manifold embedding (DTME) is proposed for discriminant analysis of medical hyperspectral images. Based on the idea of manifold learning, a new discriminant similarity metric is designed, which takes into account the tensor representation, sparsity, low-rank and distribution characteristics. Then, an inter-class tensor graph and an intra-class tensor graph are constructed using the new similarity metric to reveal intrinsic manifold of hyperspectral data. Dimensionality reduction is achieved by embedding this supervised tensor graphs into the low-dimensional tensor subspace. Experimental results on membranous nephropathy and white bloodcells identification tasks demonstrate the potential clinical value of the proposed DTME.

The traditional differential diagnosis of membranous nephropathy (MN) mainly relies on clinical symptoms, serological examination and optical renal biopsy. However, there is a probability of false positives in the optical inspection results, and it is unable to detect the change of biochemical components, which poses an obstacle to pathogenic mechanism analysis. Microscopic hyperspectral imaging can reveal detailed component information of immune complexes, but the high dimensionality of microscopic hyperspectral image brings difficulties and challenges to image processing and disease diagnosis. In this paper, a novel classification framework, including spatial-spectral density peaks-based discriminant analysis (SSDP), is proposed for intelligent diagnosis of MN using a microscopic hyperspectral pathological dataset. SSDP constructs a set of graphs describing intrinsic structure of MHSI in both spatial and spectral domains by employing density peak clustering. In the process of graph embedding, low-dimensional features with important diagnostic information in the immune complex are obtained by compacting the spatial-spectral local intra-class pixels while separating the spectral inter-class pixels. For the MN recognition task, a support vector machine (SVM) is used to classify pixels in the low-dimensional space. Experimental validation data employ two types of MN that are difficult to distinguish with optical microscope, including primary MN and hepatitis B virus-associated MN. Experimental results show that the proposed SSDP achieves a sensitivity of 99.36%, which has potential clinical value for automatic diagnosis of MN.

Collaborative Representation Detection (CRD) is a very effective anomaly detection method, which is directly based on the concept that pixel under test (PUT) can be approximately linear represented by its spatial adjacent background pixels. If the adjacent background pixels are contaminated, the approximate value of PUT linearly represented by the surrounding pixels is inaccurate. In this work, an improved method for anomaly detection in hyperspectral imagery is proposed based on CRD. In our proposed method, the least squares technique first is adopted to obtain the preliminary linear representation coefficient, which is positively correlated with its contribution to PUT. Then, the purified background pixels are obtained according to the numerical value of the representation coefficient. Generally, the anomaly pixels are usually different from the background pixels, so saliency weight is imposed on the test pixel to make full use of the spatial information of inner window pixels around the test pixel. Extensive experiments for real hyperspectral datasets show that the proposed method outperforms the CRD method and other traditional detection methods.

Classifying different land cover types with hyperspectral image (HSI) is significant for restoring and protecting natural resources and maintaining ecological services in coastal wetlands. This paper proposes a multi-domain features fusion convolutional neural network (MDF-CNN) based classification method for hyperspectral images of coastal wetlands. This method adopts inter-class sparsity based discriminative least square regression (ICS_DLSR) to learn a more compact and discriminative transformation, as well as fuse the high-level features of the original domain and the regression domain to obtain higher classification accuracy. Experimental results demonstrate the effectiveness of the proposed method when compared with some recent classifiers. The MDF-CNN achieved state-of-the-art performance on two latest GF-5 HSI datasets of Coastal Wetland.

Target detection in hyperspectral images (HSI) is an important technique and many target detection algorithms have been developed in recent years. The most widely detection algorithms by the original spectral characteristics may lack the ability of target signal enhancement and background suppression. This paper presents an efficient algorithm for detecting hyperspectral targets based on fractional Fourier transform (FrFT). Firstly, fractional Fourier transform primary search is used as preprocessing to obtain the better intermediate domain features with complementary characteristics between the original reflection spectrum and the Fourier transform domain. Secondly, fractional Fourier transform secondary search and constrained energy minimization (FrFT-CEM) was adopted to find an optimal fractional order to distinguish the target from the background. The proposed method has been proved to be superior in two real hyperspectral data sets.

Sea-ice classification plays an important role in evaluating sea-ice hazards and ensuring maritime safety. In this paper, a method of sea-ice classification based on morphological feature extraction is proposed by using CEBRS-02B multispectral CCD image. The paper uses local contain profile (LCP) to extract morphological features of the multispectral image. Then SVM- and MRF- (SVMMRF) is adopted, which including probabilistic support vector machine (SVM) for the preliminary classification of multispectral image, the postprocessing by using Markov random field (MRF) based regularization. Experimental results demonstrate the validity of the classification model framework is applied to sea-ice classification. Compared with the traditional Local Binary Pattern (LBP) and Gabor, feature extraction by LCP can improve the accuracy of sea-ice classification.

Multisensor fusion is of great importance in Earth observation related applications. For instance, hyperspectral images (HSIs) provide wealthy spectral information while light detection and ranging (LiDAR) data provide elevation information, and using HSI and LiDAR data together can achieve better classification performance. In this paper, an unsupervised feature extraction framework, named as patch-to-patch convolutional neural network (PToP CNN), is proposed for collaborative classification of hyperspectral and LiDAR data. More specific, a three-tower PToP mapping is first developed to seek an accurate representation from HSI to LiDAR data, aiming at merging multiscale features between two different sources. Then, by integrating hidden layers of the designed PToP CNN, extracted features are expected to possess deeply fused characteristics. Accordingly, features from different hidden layers are concatenated into a stacked vector and fed into three fully connected layers. To verify the effectiveness of the proposed classification framework, experiments are executed on two benchmark remote sensing data sets. The experimental results demonstrate that the proposed method provides superior performance when compared with some state-of-the-art classifiers, such as two-branch CNN and context CNN.

Deep learning methods have been widely used in hyperspectral image classification and have achieved state-of-the-art performance. Nonetheless, the existing deep learning methods are restricted by a limited receptive field, inflexibility, and difficult generalization problems in hyperspectral image classification. To solve these problems, we propose HSI-BERT, where BERT stands for bidirectional encoder representations from transformers and HSI stands for hyperspectral imagery. The proposed HSI-BERT has a global receptive field that captures the global dependence among pixels regardless of their spatial distance. HSI-BERT is very flexible and enables the flexible and dynamic input regions. Furthermore, HSI-BERT has good generalization ability because the jointly trained HSI-BERT can be generalized from regions with different shapes without retraining. HSI-BERT is primarily built on a multihead self-attention (MHSA) mechanism in an MHSA layer. Moreover, several attentions are learned by different heads, and each head of the MHSA layer encodes the semantic context-aware representation to obtain discriminative features. Because all head-encoded features are merged, the resulting features exhibit spatial-spectral information that is essential for accurate pixel-level classification. Quantitative and qualitative results demonstrate that HSI-BERT outperforms any other CNN-based model in terms of both classification accuracy and computational time and achieves state-of-the-art performance on three widely used hyperspectral image data sets.

In recent years, deep learning has presented a great advance in the hyperspectral image (HSI) classification. Particularly, long short-term memory (LSTM), as a special deep learning structure, has shown great ability in modeling long-term dependencies in the time dimension of video or the spectral dimension of HSIs. However, the loss of spatial information makes it quite difficult to obtain better performance. In order to address this problem, two novel deep models are proposed to extract more discriminative spatial-spectral features by exploiting the convolutional LSTM (ConvLSTM). By taking the data patch in a local sliding window as the input of each memory cell band by band, the 2-D extended architecture of LSTM is considered for building the spatial-spectral ConvLSTM 2-D neural network (SSCL2DNN) to model long-range dependencies in the spectral domain. To better preserve the intrinsic structure information of the hyperspectral data, the spatial-spectral ConvLSTM 3-D neural network (SSCL3DNN) is proposed by extending LSTM to the 3-D version for further improving the classification performance. The experiments, conducted on three commonly used HSI data sets, demonstrate that the proposed deep models have certain competitive advantages and can provide better classification performance than the other state-of-the-art approaches.

Least-squares regression (LSR)-based classifiers are effective in multiclassification tasks. However, most existing methods use limited projections, resulting in loss of much discriminant information; furthermore, they focus only on exactly fitting samples to target matrix while ignoring overfitting issue. To solve these drawbacks, discriminative marginalized LSR (DMLSR) is proposed to learn a more discriminative projection matrix with consideration of class separability and data-reconstruction ability simultaneously. In the proposed framework, an intraclass compactness graph is employed to avoid the overfitting problem and enhance class separability, and a data-reconstruction constraint is imposed to preserve discriminant information on limited projections. Experimental results on several hyperspectral data sets demonstrate that the proposed method significantly outperforms some state-of-the-art classifiers.

Earth observation using multisensor data is drawing increasing attention. Fusing remotely sensed hyperspectral imagery and light detection and ranging (LiDAR) data helps to increase application performance. In this article, joint classification of hyperspectral imagery and LiDAR data is investigated using an effective hierarchical random walk network (HRWN). In the proposed HRWN, a dual-tunnel convolutional neural network (CNN) architecture is first developed to capture spectral and spatial features. A pixelwise affinity branch is proposed to capture the relationships between classes with different elevation information from LiDAR data and confirm the spatial contrast of classification. Then in the designed hierarchical random walk layer, the predicted distribution of dual-tunnel CNN serves as global prior while pixelwise affinity reflects the local similarity of pixel pairs, which enforce spatial consistency in the deeper layers of networks. Finally, a classification map is obtained by calculating the probability distribution. Experimental results validated with three real multisensor remote sensing data demonstrate that the proposed HRWN significantly outperforms other state-of-the-art methods. For example, the two branches CNN classifier achieves an accuracy of 88.91% on the University of Houston campus data set, while the proposed HRWN classifier obtains an accuracy of 93.61%, resulting in an improvement of approximately 5%.

Cell classification, especially that of white blood cells, plays a very important role in the field of diagnosis and control of major diseases. Compared to traditional optical microscopic imaging, hyperspectral imagery, combined with both spatial and spectral information, provides more wealthy information for recognizing cells. In this paper, a novel blood cell classification framework, which combines a modulated Gabor wavelet and deep convolutional neural network (CNN) kernels, named as MGCNN, is proposed based on medical hyperspectral imaging. For each convolutional layer, multi-scale and orientation Gabor operators are taken dot product with initial CNN kernels. The essence is to transform the convolutional kernels into the frequency domain to learn features. By combining characteristics of Gabor wavelets, the features learned by modulated kernels at different frequencies and orientations are more representative and discriminative. Experimental results demonstrate that the proposed model can achieve better classification performance than traditional CNNs and widely used support vector machine approaches, especially as training small-sample-size situations.

Due to the limitation of image acquisition, hyperspectral remote sensing imagery is hard to reflect in both high spatial and spectral resolutions. Super-resolution (SR) is a technique which can improve the spatial resolution. Inspired by recent achievements in deep convolutional neural network (CNN) and generative adversarial network (GAN), a GAN based framework is proposed for hyperspectral image super-resolution. In the proposed method, residual learning is used to obtain a high metrics and spectral fidelity, and a shorter connection is set between the input layer and output layer. The gradient features from low-resolution (LR) image to high-resolution (HR) are utilized as auxiliary information to assist deep CNN to carry out counter training with discriminator. Experimental results demonstrate that the proposed SR algorithm achieves superior performance in spectral fidelity and spatial resolution compared with baseline methods.

Collaborative classification of hyperspectral imagery (HSI) and light detection and ranging (LiDAR) data is investigated using effective hierarchical random walk networks, denoted as HRWN. The proposed HRWN jointly optimizes dual-tunnel CNN, pixelwise affinity and seeds map via a novel random walk layer, which enforces spatial consistency in the deepest layers of the network. In designed random walk layer, the predicted distribution of dual-tunnel CNN serves as global prior while pixelwise affinity reflects local similarity of pixel pairs, which preserves boundary localization and spatial consistency well. Experimental results validated with two real multisource remote sensing data demonstrate that the proposed HRWN can significantly outperform other state-of-art methods.

Convolutional neural network (CNN) receives extensive attention in hyperspectral image classification. While hyper-spectral images contain abundant spectral information but lack spatial information, which usually contributes to poor classification results. In this paper, a novel classification framework called information fusion based CNN (IF-CNN) is proposed to compensate for the shortcomings of hyper-spectral images. The proposed method merges hyperspectral images with abundant spectral information and LiDAR images with rich spatial information as the input of classification framework. Furthermore, the framework consists of two convolutional neural networks: one-dimensional CNN for extracting spectral features, and two-dimensional CNN for extracting spatial correlation features. Experimental results demonstrate that the proposed method achieves excellent performance compared with some existing methods.

For obtaining information about ecosystem resource, GF-1 satellite was launched on April 26, 2013, which is the first satellite of the China's High-Resolution Earth Observation System. After obtaining some of the remote sensing images from GF-1, we selected WFV(wide field vision) images and detected the woodland to separate it from other geography types in the images. First, WFV images were clipped and labeled, then two deep learning models, POI-Net and Deep-UNet were used for training. We fused the prediction matrixes of deep learning networks using proposed "Most-sure strategy". The results show that our method can effectively improve the accuracy of woodland detection and segmentation results are outstanding. In addition, the proposed framework can also detect woodland in images returned by GF-6 satellite.

The fusion of Panchromatic (PAN) and Hyperspectral image (HSI) aims at improving resolution in spatial and spectral domain simultaneously. Multiresolution analysis is a widely used method for Mutispectral or HSI pansharpening. However, only detail information from PAN is considered while ignoring the detail information from HSI. In this paper, an improved approach based on multiresolution analysis is proposed, which extracts detail information from both PAN and HSI by choosing optimal multiresolution layers. Another contribution is that we discuss the weight when fusing the detail information. The experimental results demonstrate that the proposed method can provide better quality metrics and visual effects when compared with some existing methods.

Sparse and low rank matrix decomposition is a method that has recently been developed for estimating different components of hyperspectral data. The rank component is capable of preserving global data structures of data, while a sparse component can select the discriminative information by preserving details. In order to take advantage of both, we present a novel decision fusion based on joint low rank and sparse component (DFJLRS) method for hyperspectral imagery in this paper. First, we analyzed the effects of different components on classification results. Then a novel method adopts a decision fusion strategy which combines a SVM classifier with the information provided by joint sparse and low rank components. With combination of the advantages, the proposed method is both representative and discriminative. The proposed algorithm is evaluated using several hyperspectral images when compared with traditional counterparts.

In this paper, we propose to extract the multi-scaled and rotation-insensitive deep features to address the issues of object multi-solutions and rotations in geospatial object detection. To this end, we develop a novel object detection framework where a rotation-insensitive convolution neural network is applied for extracting multi-scaled and direction-insensitive feature representation and then the learned features can be fed into the ensemble classifier learning with fast feature pyramid. Such a non-end-to-end learning strategy intuitively reduces the computational cost without the additional performance loss, yielding an effective and efficient light-weight object detection framework. Experimental results conducted on the NWPU VHR-10 dataset demonstrate that the proposed framework outperforms several state-of-the-art baselines.

Infrared (IR) small target detection of low signal-to-noise ratio (SNR) is a very meaningful and challenging subject in detecting and tracking system. Therefore, an effective method is proposed in this paper. First of all, a morphological feature extraction method is used to reconstruct a new image that the small target is disappeared, then the original image and the reconstruction image are made difference as preprocessed image. In this way, important spatial information can be extracted well in IR image. Then, a low-rank and sparse decomposition method is employed to obtain the background image and the target image respectively, the target separation can be enhanced and the background clutter can be suppressed simultaneously. Finally, the obtained target image is segmented by a simple adaptive segmentation method. The experimental results indicate that the proposed method is of great improvement compared with several existing methods, what's more, it can achieve the highest SNR among these methods.

Convolutional neural network (CNN) has been widely used in hyperspectral imagery (HSI) classification. Data augmentation is proven to be quite effective when training data size is relatively small. In this letter, extensive comparison experiments are conducted with common data augmentation methods, which draw an observation that common methods can produce a limited and up-bounded performance. To address this problem, a new data augmentation method, named as pixel-block pair (PBP), is proposed to greatly increase the number of training samples. The proposed method takes advantage of deep CNN to extract PBP features, and decision fusion is utilized for final label assignment. Experimental results demonstrate that the proposed method can outperform the existing ones.

This letter presents an efficient probabilistic collaborative representation-based classifier (PROCRC) for hyperspectral image classification. Its performance is evaluated on different types of spatial features of hyperspectral imagery (HSI) including shape feature (i.e., extended multiattribute feature), global feature (i.e., Gabor feature), and local feature [i.e., local binary pattern (LBP)]. Compared with the original collaborative representation classifier (CRC), the proposed PROCRC offers superior classification performance. The Tikhonov regularized versions of CRC have excellent classification performance but their computational cost is high. The experimental results show that the PROCRC can yield comparable classification accuracy but with much lower computational cost.

Human action recognition based on deep-learning methods have received increasing attention and developed rapidly. However, current methods suffer from the confusion caused by convolving over time and space independently, processing shorter sequences, restricted to single temporal scale modeling and so on. The key objective of precisely classifying actions is to capture the appearance and motion throughout entire videos. Based on this purpose, a multi-branch spatial-temporal network (MSTN) is proposed. It consists of a multi-branch deep network and a long-term feature (LTF) layer. Benefits of the proposed MSTN include: (a) the multi-branch spatial-temporal network aims at encoding spatial and temporal information simultaneously, and (b) the LTF layer is used to aggregate the video-level representation with multiple temporal scales. Evaluations on two action datasets and comparison with several state-of-the-art approaches demonstrate the effectiveness of the proposed network.

Microwave radiometers are the key sensors to globally monitor environmental parameters; however, it suffers from its low and nonuniform spatial resolution. In this paper, a superresolution (SR) technique based on image degradation and residual network is proposed to enhance the spatial resolution of microwave radiometer data. Specifically, an improved degradation model is proposed to construct pairs of high-resolution (HR) and low-resolution (LR) data for training and testing. In addition, a new residual network connected by the SR main and gradient auxiliary branches in parallel is designed to achieve SR reconstructions, where eight-channel gradient maps extracted from LR data are input into the auxiliary branch to help to reconstruct. SR results are eventually generated by the trained SR network. Experiments executed on both simulated and actual data demonstrate the soundness and the superiority of the proposed SR technique.

Recently, collaborative representation (CR) has drawn increasing attention in hyperspectral image classification due to its simplicity and effectiveness. However, existing representation-based classifiers do not explicitly utilize class label information of training samples in estimating representation coefficients. To solve this issue, a structure-aware CR with Tikhonov regularization (SaCRT) method is proposed to consider both class label information of training samples and spectral signatures of testing pixels to estimate more discriminative representation coefficients. In the proposed framework, marginal regression is employed; furthermore, an interclass row-sparsity structure is designed to preserve the compact relationship among intraclass pixels and more separable interclass pixels, thereby enhancing class separability. The experimental results evaluated using three hyperspectral data sets demonstrate that the proposed method significantly outperforms some state-of-the-art classifiers.

Pushbroom acquisition of hyperspectral imagery is prone to striping artifacts in the along-track direction. A hyperspectral destriping algorithm is proposed such that the subbands of a 3-D wavelet transform most affected by pushbroom stripes-namely, those with spatially vertical orientation-are the exclusive focus of destriping. The proposed method features an iterative image decomposition composed of a low-rank model for the stripes coupled with a group-sparse prior on the wavelet coefficients of the subbands in question. While low-rank stripe models have been widely used in the past, they typically have been deployed in conjunction with a total-variation prior on the image that is prone to oversmoothing and residual stripe artifacts. On the other hand, the proposed group-sparse prior not only captures the well-known sparse nature of wavelet coefficients but also capitalizes on their vertical clustering in the subbands in question. In addition, while many prior destriping methods are wavelet-based, they employ 2-D transforms band by band. In contrast, the proposed 3-D wavelet transform provides a greater concentration of stripe information into fewer wavelet coefficients, leading to more effective destriping. Experimental results on both synthetically striped imagery as well as real striped imagery from an actual hyperspectral sensor demonstrate superior image quality for the proposed method as compared with other state-of-the-art methods.

With the rapid development of spaceborne imaging techniques, object detection in optical remote sensing imagery has drawn much attention in recent decades. While many advanced works have been developed with powerful learning algorithms, the incomplete feature representation still cannot meet the demand for effectively and efficiently handling image deformations, particularly objective scaling and rotation. To this end, we propose a novel object detection framework, called Optical Remote Sensing Imagery detector (ORSIm detector), integrating diverse channel features extraction, feature learning, fast image pyramid matching, and boosting strategy. An ORSIm detector adopts a novel spatial-frequency channel feature (SFCF) by jointly considering the rotation-invariant channel features constructed in the frequency domain and the original spatial channel features (e.g., color channel and gradient magnitude). Subsequently, we refine SFCF using learning-based strategy in order to obtain the high-level or semantically meaningful features. In the test phase, we achieve a fast and coarsely scaled channel computation by mathematically estimating a scaling factor in the image domain. Extensive experimental results conducted on the two different airborne data sets are performed to demonstrate the superiority and effectiveness in comparison with the previous state-of-the-art methods.

Small infrared target detection is a key and challenging issue in object detection and tracking systems. Existing algorithms can be mainly categorized into nonlocal-based or local-based methods. However, the detection performance degrades rapidly when facing highly heterogeneous backgrounds. This is mainly due to that they exploit only one kind of information (e.g., local or nonlocal) while sacrificing the other. Thus, an effective small target detection method is proposed to combine local and nonlocal priors. The former is obtained by a sliding dual window while the latter is realized by low-rank and sparse decomposition. Experimental results on three real datasets validate the effectiveness of the proposed framework, which is more stable and robust compared with several state-of-the-art methods, especially for the image scenes with heavy background clutters.

Different enhanced convolutional neural network (CNN) architectures have been proposed to surpass very deep layer bottleneck by using shortcut connections. In this paper, we present an effective deep CNN architecture modified on the typical Residual Network (ResNet), named as Cascade Network (CascadeNet), by repeating cascade building blocks. Each cascade block contains independent convolution paths to pass information in the previous layer and the middle one. This strategy exposes a concept of “cross-passing” which differs from the ResNet that stacks simple building blocks with residual connections. Traditional residual building block do not fully utilizes the middle layer information, but the designed cascade block catches cross-passing information for more complete features. There are several characteristics with CascadeNet: enhance feature propagation and reuse feature after each layer instead of each block. In order to verify the performance in CascadeNet, the proposed architecture is evaluated in different ways on two data sets (i.e., CIFAR-10 and HistoPhenotypes dataset), showing better results than its ResNet counterpart.

Along with very high-resolution satellites were launched frequently, such as the satellite WorldView-3, panchromatic and multispectral remote-sensing images can be acquired easily. However, it is still an interesting and challenging task to fuse and classify these images. In general, panchromatic image has a high spatial resolution, but with only one spectral band. Multispectral image usually has four or eight bands, but the spatial resolution is four times smaller than panchromatic image. In this paper, an unsupervised feature extraction framework is proposed, which combines multispectral (MS) image and panchromatic (PAN) image into convolution neural network (CNN). There is an image-to-image mapping, learning from the input source (i.e., MS) to the output source (i.e., PAN). Then, by integrating the hidden layer of deep CNN, the extracted features represent MS and PAN data. The experimental results of two practical remote sensing data sets show the validity of the framework.

Deep convolutional neural network (CNN) is an important branch of deep learning. Due to its strong ability of feature extraction, CNN models have been introduced to solve the problems of synthetic aperture radar automatic target recognition (SAR-ATR). However, labeled SAR images are difficult to acquire. Therefore, how to obtain a good recognition result from a small sample dataset is what we mainly focus on. In theory, a deeper network can bring a better training result. But it also brings more difficulties to the training process, especially with limited labeled training data. The residual learning which proposed in recent years can alleviate this problem effectively. In this paper, we use a deep residual network, and introduce the dropout layer into the building block to alleviate overfitting caused by limited SAR data. In order to improve the training effect, the new loss function center loss is adopted and combined with softmax loss as the supervision signal to train the deep CNN. The experimental results show that our method can achieve the classification accuracy of 99.67% with all training data, without data augmentation or pre-training. When data of the training dataset was reduced to 20%, we can still achieve a recognition result higher than 94%.

This paper presents Gabor-filtering-based probabilistic collaborative representation for hyperspectral image classification. Compared with the original collaborative representation classifier (CRC) and the CRC using Gabor features, the proposed classifier offers superior classification performance. The regularized versions of CRC using Gabor features have excellent classification performance; however, those classifiers have high computational cost. Experimental results show that the proposed approach can generate high classification accuracy with lower computational cost.

Recently, the deep convolutional neural network (CNN) is of great interest in hyperspectral image classification. However, limited available training samples still prevent CNN from exploring the performance of classification. In this work, we employ a novel pixel-pair method based on Siamese neural network (SNN) to significantly enlarge the training set and better represent the spectral-spatial features. In training, two pixels are respectively fed into two branch CNNs to extract deep features, where the same weights and biases are shared. Then, the absolute difference between the two deep features is learned by linear full connection layers with a given label. In testing, pixel-pairs, constructed by combining the center pixel and each of the surrounding pixels, are classified by the trained SNN. The final prediction is then determined by a voting strategy. The proposed SNN framework is extended to learn deep patch-pixel features. Experimental performance demonstrates that the proposed strategy outperforms the traditional classifiers, such as support vector machine (SVM) and extreme learning machine (ELM).

Conventional classification algorithms have shown great success for balanced classes. In remote sensing applications, it is often the case that classes are imbalanced. This paper proposes a novel solution to solve the problem of imbalanced training samples in hyperspectral image classification. It consists of two parts: one is for large-size sample sets and the other is for small-size sets. Specifically, an algorithm based on the orthogonal complement subspace projection (OCSP) is proposed to select samples from large-size classes, and an algorithm also based on OCSP is proposed to create artificial samples for small-size ones. The impact on representation-based classifiers, i.e., sparse and collaborative representation classifiers and traditional classifiers (e.g., support vector machine), is investigated. Experimental results demonstrate that the proposed solution can outperform other existing solutions in the literature.

As a list of remotely sensed data sources is available, how to efficiently exploit useful information from multisource data for better Earth observation becomes an interesting but challenging problem. In this paper, the classification fusion of hyperspectral imagery (HSI) and data from other multiple sensors, such as light detection and ranging (LiDAR) data, is investigated with the state-of-the-art deep learning, named the two-branch convolution neural network (CNN). More specific, a two-tunnel CNN framework is first developed to extract spectral-spatial features from HSI; besides, the CNN with cascade block is designed for feature extraction from LiDAR or high-resolution visual image. In the feature fusion stage, the spatial and spectral features of HSI are first integrated in a dual-tunnel branch, and then combined with other data features extracted from a cascade network. Experimental results based on several multisource data demonstrate the proposed two-branch CNN that can achieve more excellent classification performance than some existing methods.

Graph embedding, as a dimensionality reduction framework, has already drawn great attention in hyperspectral image analysis. Taking locality preserving projection (LPP) as example, LPP utilizes typical Euclidean distance in heat kernel to create an affinity matrix and projects the high-dimensional data into a lower-dimensional space. However, the Euclidean distance is not sufficiently correlated with intrinsic spectral variation of a material, which may result in inappropriate graph representation. In this work, a graph-based discriminant analysis with novel spectral similarity measurement is proposed, which fully considers curves changing description among spectral bands. Experimental results based on real hyperspectral images demonstrate the proposed method is superior to traditional methods, such as supervised LPP, and the state-of-the-art sparse graph-based discriminant analysis (SGDA).

An interesting target detection framework with transferred deep convolutional neural network (CNN) is proposed. For CNN, many labeled samples are needed to train the multi-layer network. However, for target detection tasks, only few target spectral signatures are available, or they are unknown in anomaly detection. In this work, we employ a reference data and further generate pixel-pairs to enlarge the sample size. A multi-layer CNN is trained by using difference between pixel-pairs generated from the reference image scene. During testing, there are two cases: (1) for anomaly detection, difference between pixel-pairs, constructed by combing the center pixel and its surrounding pixels, is classified by the trained CNN with result of similarity measurement; and (2) for supervised target detection, difference between pixel-pairs, constructed by combing the testing pixel and the known spectral signatures, is classified. The detection output is simply generated by averaging these similarity scores. Experimental performance demonstrates that the proposed strategy outperforms the classic detectors.

The paper investigates representation-based classification for multispectral imagery. Due to the limited spectral dimension, the performance may be limited, and, in general, it is difficult to discriminate different classes using multispectral imagery. Nonlinear band generation method is proposed to use which can provide additional spectral information for multispectral classification. Two classifiers, sparse representation-based classification (SRC) and Nearest Regularized Subspace (NRS) are evaluated on the generated datasets. The results show our approach can outperform other nonlinear method such as the traditional kernel method in terms of classification accuracy and computational cost.

Synthetic Aperture Radar(SAR) has been widely used in airborne remote sensing and satellite ocean observation fields to reduce the affect of weather condition and sun illumination. As technology developed, swath and resolution requirements are increased in terrain, which arouse a huge increase in the number of simulated points and simulated pulses and lead to a huge increase in simulated time. With the development of graphics processing unit(GPU), it can parallel simulated points to reduce simulated time. As for increased simulated pulses, they can be paralleled on distributed computers. In the article, we focus on parallel on increased simulated pulses and put forward two frameworks based on message passing libraries (MPI) and cloud computing (Hadoop).

In this letter, a novel anomaly detection framework with transferred deep convolutional neural network (CNN) is proposed. The framework is designed by considering the following facts: 1) a reference data with labeled samples are utilized, because no prior information is available about the image scene for anomaly detection and 2) pixel pairs are generated to enlarge the sample size, since the advantage of CNN can be realized only if the number of training samples is sufficient. A multilayer CNN is trained by using difference between pixel pairs generated from the reference image scene. Then, for each pixel in the image for anomaly detection, difference between pixel pairs, constructed by combining the center pixel and its surrounding pixels, is classified by the trained CNN with the result of similarity measurement. The detection output is simply generated by averaging these similarity scores. Experimental performance demonstrates that the proposed algorithm outperforms the classic Reed-Xiaoli and the state-of-the-art representation-based detectors, such as sparse representation-based detector (SRD) and collaborative representation-based detector.

In this letter, a low-rank and sparse representation classifier with a spectral consistency constraint (LRSRC-SCC) is proposed. Different from the SRC that represents samples individually, LRSRC-SCC reconstructs samples jointly and is able to capture the local and global structures simultaneously. In this proposed classifier, an adaptive spectral constraint is imposed on both the low-rank and sparse terms so as to better reveal the data structure and enhance its discriminative power. In addition, the alternating direction method is introduced to solve the underlying minimization problem, in which, more importantly, the subobjective function associated with the low-rank term is optimized based on the rank equivalence between a matrix and its Gram matrix, resulting in a closed-form solution. Finally, LRSRC-SCC is extended to LRSRC-SCCE for fully exploiting the spatial information. Experimental results on two hyperspectral data sets demonstrate that the proposed LRSRC-SCC and LRSRC-SCCE methods outperform some state-of-the-art methods.

This letter proposes to integrate the locality sensitive discriminant analysis (LSDA) with the group sparse representation (GSR) for a hyperspectral imagery classification. The LSDA is to project the data set to a lower-dimensional subspace to preserve local manifold structure and discriminant information, while the GSR is to encode the projected testing set as a sparse linear combination of group-structured training samples for classification. The proposed approach, denoted as LSDA-GSR classifier (GSRC), is evaluated using two real hyperspectral data sets. Experimental results demonstrate that it can provide considerable improvement to the original counterparts, i.e., SRC and GSRC, with a relatively low computational cost.

The deep convolutional neural network (CNN) is of great interest recently. It can provide excellent performance in hyperspectral image classification when the number of training samples is sufficiently large. In this paper, a novel pixel-pair method is proposed to significantly increase such a number, ensuring that the advantage of CNN can be actually offered. For a testing pixel, pixel-pairs, constructed by combining the center pixel and each of the surrounding pixels, are classified by the trained CNN, and the final label is then determined by a voting strategy. The proposed method utilizing deep CNN to learn pixel-pair features is expected to have more discriminative power. Experimental results based on several hyperspectral image data sets demonstrate that the proposed method can achieve better classification performance than the conventional deep learning-based method.

Due to the high-dimensional characteristic of hyperspectral images, dimensionality reduction (DR) is an important preprocessing step for classification. Recently, sparse and low-rank graph-based discriminant analysis (SLGDA) has been developed for DR of hyperspectral images, for which the properties of sparsity and low-rankness are simultaneously exploited to capture both local and global structures. However, SLGDA may not achieve satisfactory results when handling complex data with nonlinear nature. To address this problem, this paper presents two kernel extensions of SLGDA. In the first proposed classical kernel SLGDA (cKSLGDA), the kernel trick is exploited to implicitly map the original data into a high-dimensional space. With a totally different perspective, we further propose a Nyström-based kernel SLGDA (nKSLGDA) by constructing a virtual kernel space by the Nyström method, in which virtual samples can be explicitly obtained from the original data. Both cKSLGDA and nKSLGDA can achieve more informative graphs than SLGDA, and offer superiority over other state-of-the-art DR methods. More importantly, the nKSLGDA can outperform cKSLGDA with much lower computational cost.

In this paper, convolutional neural networks (CNNs) is employed for remote-sensing scene classification, which fully utilizes the semantic features extracted from the images while ignoring some traditional features. Consider the limited labeled samples, CaffeNet model as the pre-trained architecture is adopted. By fine-tuning the pre-trained models, the proposed method is expected to be robust and efficient. Its performance is evaluated with two remote-sensing scene datasets. From the experimental results, the proposed CNN-based scene classification method does provide more excellent performance and be superior to several state-of-the-art methods.

This paper proposes a novel solution to solve the problem of imbalanced training samples in hyperspectral image classification. It consists of two parts: one is for large-size sample sets and the other is for small-size sets. We exploit an orthogonal projection based algorithm to select samples from large-size ones; meanwhile, we propose an algorithm based on the orthogonal complementary subspace projection to create artificial samples for small-size ones. The impact on representation based classifiers, i.e., sparse representation based classifier and collaborative representation based classifier, are investigated. Experimental results demonstrate that it can outperform other traditional solutions.

Deep convolutional neural networks (CNNs) have brought in achievements in image classification and target detection. In this paper, we propose a novel five-layer CNN for hyperspectral classification by encountering recent achievement in deep learning area, such as batch normalization, dropout, Parametric Rectified Linear Unit (PReLu) activation function. By taking advantage of the specific characteristics of hyperspectral images, spatial context and spectral information are elegantly integrated into the framework. Experimental results demonstrate that our proposed CNN out- performs the state-of-the-art methods.

This paper presents a new simulator for spaceborne SAR on-board imaging process on mobile GPUs. The system can generate raw data and perform imaging process in real time. Due to introducing low power GPUs, it has low power consumption, light weight and high computing capability. This simulator has the unique structure, which can guarantee its real-time processing. The experimented results indicate that it is possible to apply the simulator to simulating spaceborne SAR imaging process, and this portable simulator can be used in other areas, which could broaden new horizons in low power space.

Collaborative representation-based classification with distance-weighted Tikhonov regularization (CRT) has offered high accuracy and efficiency. Due to its per-pixel classification nature without a training step, this paper develops a parallel implementation by using compute unified device architecture (CUDA) on graphics processing units (GPUs). To further improve classification accuracy, local binary pattern (LBP) is used for spatial feature extraction, and an unsupervised band selections approach is applied for dimensionality reduction and an optimized collaborative model combining spatial-spectral features is employed. The proposed parallel implementation is able to increase computational efficiency while not degrading classification accuracy when compared with the serial implementations on central processing units (CPUs).

In this paper, we address the problem of ship detection in PolSAR image. We firstly investigate the differences of scattering mechanism between ship targets and the sea surface based on the polarimetric similarity analysis. It is shown that, the sea surface scattering is dominated by the odd bounce (denoted as r1), while the scattering of ship targets are both dominated by the even bounce scattering (r2) which has been widely accepted, and the line bounce scattering (r4) which is a new find from the experiments. Based on those differences, a metric (r2+r4)/r1can be applied to distinguish ship targets from the sea surface. To further suppress the effects of sidelobes and imaging artifacts which have the same/similar scattering behaviors with ship targets, we introduce in the third eigenvalue λ3) of the coherency matrix and obtain the metric (r2+ r4)λ3/r1. The preliminary results show that a constant false alarm rate (CFAR) ship detector based on the proposed metric can obtain promising ship detection performance.

Synthetic Aperture Radar (SAR) has been widely used in airborne remote sensing and satellite ocean observation fields to reduce the affect of weather condition and sun illumination. As technology developed, swath and resolution requirements are increased in terrain, which result in a huge increase in echo data and simulated time[1]. With the development of graphics processing unit (GPU), it can reduce simulated time effectively. In order to simulate the coherent integration, atomic operation is always used in GPU, which has a bad influence to simulated time. To optimize simulated time, in this article, we put forward three GPU optimistic strategies for atomic-free SAR raw data simulation.

Collaborative graph-based discriminant analysis (CGDA) has been recently proposed for dimensionality reduction and classification of hyperspectral imagery, offering superior performance. In CGDA, a graph is constructed by ℓ2- norm minimization-based representation using available labeled samples. Different from sparse graph-based discriminant analysis (SGDA) where a graph is built by ℓ1- norm minimization, CGDA benefits from within-class sample collaboration and computational efficiency. However, CGDA does not consider data manifold structure reflecting geometric information. To improve CGDA in this regard, a Laplacian regularized CGDA (LapCGDA) framework is proposed, where a Laplacian graph of data manifold is incorporated into the CGDA. By taking advantage of the graph regularizer, the proposed method not only can offer collaborative representation but also can exploit the intrinsic geometric information. Moreover, both CGDA and LapCGDA are extended into kernel versions to further improve the performance. Experimental results on several different multiple-class hyperspectral classification tasks demonstrate the effectiveness of the proposed LapCGDA.

Recently, sparse graph-based discriminant analysis (SGDA) has been developed for the dimensionality reduction and classification of hyperspectral imagery. In SGDA, a graph is constructed by ℓ1-norm optimization based on available labeled samples. Different from traditional methods (e.g., k-nearest neighbor with Euclidean distance), weights in an ℓ1-graph derived via a sparse representation can automatically select more discriminative neighbors in the feature space. However, the sparsity-based graph represents each sample individually, lacking a global constraint on each specific solution. As a consequence, SGDA may be ineffective in capturing the global structures of data. To overcome this drawback, a sparse and low-rank graph-based discriminant analysis (SLGDA) is proposed. Low-rank representation has been proved to be capable of preserving global data structures, although it may result in a dense graph. In SLGDA, a more informative graph is constructed by combining both sparsity and low rankness to maintain global and local structures simultaneously. Experimental results on several different multiple-class hyperspectral-classification tasks demonstrate that the proposed SLGDA significantly outperforms the state-of-the-art SGDA.

Due to independence of weather condition and sun illumination, Synthetic Aperture Radar (SAR) has been widely used in airborne remote sensing and satellite ocean observation fields. With the increase in swath and higher resolution requirements, SAR imaging algorithms require further study. However, as a research support, echo data has massive increase. So the high efficient echo simulation is required emergently. SAR echo simulation is optimized to accelerate, based on AVX/SSE of vector instruction set and OpenMP. Experiments demonstrate that in the case of using OpenMP and AVX / SSE combined, CPU simulation efficiency is improved by about 23 times, compared to the traditional one.

In hyerspectral remote sensing community, sparse representation based classification (SRC) is a novel concept - a testing pixel is linearly represented by labeled data, and weight coefficients are often solved by an ℓ1-norm minimization. In this work, an extension of SRC is proposed by imposing an adaptive similarity measurement between the testing pixel and labeled data on the ℓ1-norm penalty, named as adaptive SRC (ASRC). ASRC generates more discriminative sparse codes which can represent the testing pixel more robustly. Experimental results demonstrate that the proposed ASRC outperforms the traditional SRC-based classification.

With the development of synthetic aperture radar (SAR) technology in recent years, we have to face the huge amount of data. So, the fast image processing technology seems to be really important in the domain of SAR. Based on the situation above, a new method to accelerate SAR image processing is proposed in this paper. The proposed method employs SIMD (Single Instruction Multiple Data) instructions and OpenMP (Open Multiprocessing) technology on multi-core SIMD CPU to realize parallel optimization on image processing algorithms. The CS (Chirp Scaling) algorithm is also chosen to process SAR data in our experiment. The experimental results demonstrate that the proposed SIMD based implementation is able to increase performance more than 20 times when compared to the single-core baseline used.

In this letter, kernel collaborative representation with Tikhonov regularization (KCRT) is proposed for hyperspectral image classification. The original data is projected into a high-dimensional kernel space by using a nonlinear mapping function to improve the class separability. Moreover, spatial information at neighboring locations is incorporated in the kernel space. Experimental results on two hyperspectral data prove that our proposed technique outperforms the traditional support vector machines with composite kernels and other state-of-the-art classifiers, such as kernel sparse representation classifier and kernel collaborative representation classifier.

Novel collaborative representation (CR)-based nearest neighbor (NN) algorithms are proposed for hyperspectral image classification. The proposed methods are based on a CR computed by an ℓ2-norm minimization with a Tikhonov regularization matrix. More specific, a testing sample is represented as a linear combination of all the training samples, and the weights for representation are estimated by an ℓ2-norm minimization-derived closed-form solution. In the first strategy, the label of a testing sample is determined by majority voting of those with k largest representation weights. In the second strategy, local within-class CR is considered as an alternative, and the testing sample is assigned to the class producing the minimum representation residual. The experimental results show that the proposed algorithms achieve better performance than several previous algorithms, such as the original k-NN classifier and the local mean-based NN classifier.

Recently, representation-based classifiers have gained increasing interest in hyperspectral image (HSI) classification. In this letter, based on our previously developed joint collaborative representation (JCR) classifier, an improved version, which is called weighted JCR (WJCR) classifier, is proposed. JCR adopts the same weights when extracting spatial and spectral features from surrounding pixels. Differing from JCR, WJCR attempts to utilize more appropriate weights by considering the similarity between the center pixel and its surroundings. Experimental results using two real HSIs demon strate that the proposed WJCR outperforms the original JCR and some other traditional classifiers, such as the support vector machine (SVM), the SVM with a composite kernel, and simultaneous orthogonal matching pursuit.

In this letter, a sparse representation-based nearest neighbor (SRNN) classifier is proposed. Unlike the traditional k-nearest neighbor (NN) classifier that employs the Euclidean distance as similarity metric, the proposed SRNN considers sparse coefficients to determine the label of testing samples, since sparse coefficients can reflect the similarity between data and provide more discriminative information. A local SRNN (LSRNN) classifier is also proposed to utilize class-specific sparse coefficients to improve the performance. Furthermore, due to the fact that neighboring pixels tend to belong to the same class with high probability, a spatially joint version of LSRNN, called JSRNN, is developed to further improve LSRNN. The proposed SRNN, LSRNN, and JSRNN have been validated on several hyperspectral remote sensing image data sets. Experimental results demonstrate that the proposed classifiers increase the classification accuracy compared with the traditional k-NN, local mean-based NN (LMNN) classifiers, and original sparse representation classifiers using representation residuals.

It is of great interest in exploiting texture information for classification of hyperspectral imagery (HSI) at high spatial resolution. In this paper, a classification paradigm to exploit rich texture information of HSI is proposed. The proposed framework employs local binary patterns (LBPs) to extract local image features, such as edges, corners, and spots. Two levels of fusion (i.e., feature-level fusion and decision-level fusion) are applied to the extracted LBP features along with global Gabor features and original spectral features, where feature-level fusion involves concatenation of multiple features before the pattern classification process while decision-level fusion performs on probability outputs of each individual classification pipeline and soft-decision fusion rule is adopted to merge results from the classifier ensemble. Moreover, the efficient extreme learning machine with a very simple structure is employed as the classifier. Experimental results on several HSI data sets demonstrate that the proposed framework is superior to some traditional alternatives.

In this paper, collaborative representation is proposed for anomaly detection in hyperspectral imagery. The algorithm is directly based on the concept that each pixel in background can be approximately represented by its spatial neighborhoods, while anomalies cannot. The representation is assumed to be the linear combination of neighboring pixels, and the collaboration of representation is reinforced by l2-norm minimization of the representation weight vector. To adjust the contribution of each neighboring pixel, a distance-weighted regularization matrix is included in the optimization problem, which has a simple and closed-form solution. By imposing the sum-to-one constraint to the weight vector, the stability of the solution can be enhanced. The major advantage of the proposed algorithm is the capability of adaptively modeling the background even when anomalous pixels are involved. A kernel extension of the proposed approach is also studied. Experimental results indicate that our proposed detector may outperform the traditional detection methods such as the classic Reed-Xiaoli (RX) algorithm, the kernel RX algorithm, and the state-of-the-art robust principal component analysis based and sparse-representation-based anomaly detectors, with low computational cost.

The Gaussian mixture model is a well-known classification tool that captures non-Gaussian statistics of multivariate data. However, the impractically large size of the resulting parameter space has hindered widespread adoption of Gaussian mixture models for hyperspectral imagery. To counter this parameter-space issue, dimensionality reduction targeting the preservation of multimodal structures is proposed. Specifically, locality-preserving nonnegative matrix factorization, as well as local Fisher's discriminant analysis, is deployed as preprocessing to reduce the dimensionality of data for the Gaussian-mixture-model classifier, while preserving multimodal structures within the data. In addition, the pixel-wise classification results from the Gaussian mixture model are combined with spatial-context information resulting from a Markov random field. Experimental results demonstrate that the proposed classification system significantly outperforms other approaches even under limited training data.

The same high dimensionality of hyperspectral imagery that facilitates detection of subtle differences in spectral response due to differing chemical composition also hinders the deployment of traditional statistical pattern-classification procedures, particularly when relatively few training samples are available. Traditional approaches to addressing this issue, which typically employ dimensionality reduction based on either projection or feature selection, are at best suboptimal for hyperspectral classification tasks. A divide-and-conquer algorithm is proposed to exploit the high correlation between successive spectral bands and the resulting block-diagonal correlation structure to partition the hyperspectral space into approximately independent subspaces. Subsequently, dimensionality reduction based on a graph-theoretic locality-preserving discriminant analysis is combined with classification driven by Gaussian mixture models independently in each subspace. The locality-preserving discriminant analysis preserves the potentially multimodal statistical structure of the data, which the Gaussian mixture model classifier learns in the reduced-dimensional subspace. Experimental results demonstrate that the proposed system significantly outperforms traditional classification approaches, even when few training samples are employed.

In this letter, a fusion-classification system is proposed to alleviate ill-conditioned distributions in hyperspectral image classification. A windowed 3-D discrete wavelet transform is first combined with a feature grouping-a wavelet-coefficient correlation matrix (WCM)-to extract and select spectral-spatial features from the hyperspectral image dataset. The adjacent wavelet-coefficient subspaces (from the WCM) are intelligently grouped such that correlated coefficients are assigned to the same group. Afterwards, a multiclassifier decision-fusion approach is employed for the final classification. The performance of the proposed classification system is assessed with various classifiers, including maximum-likelihood estimation, Gaussian mixture models, and support vector machines. Experimental results show that with the proposed fusion system, independent of the classifier adopted, the proposed classification system substantially outperforms the popular single-classifier classification paradigm under small-sample-size conditions and noisy environments.

The one-against-one (OAO) strategy is commonly employed with classifiers-such as support vector machines-which inherently provide binary two-class classification in order to handle multiple classes. This OAO strategy is introduced for the classification of hyperspectral imagery using discriminant analysis within kernel-induced feature spaces, producing a pair of algorithms-kernel discriminant analysis and kernel local Fisher discriminant analysis-for dimensionality reduction, which are followed by a quadratic Gaussian maximum-likelihood-estimation classifier. In the proposed approach, a multiclass problem is broken down into all possible binary classifiers, and various decision-fusion rules are considered for merging results from this classifier ensemble. Experimental results using several hyperspectral data sets demonstrate the benefits of the proposed approach-in addition to improved classification performance, the resulting classifier framework requires reduced memory for estimating kernel matrices.

Reconstruction of hyperspectral imagery from spectral random projections is considered. Specifically, multiple predictions drawn for a pixel vector of interest are made from spatially neighboring pixel vectors within an initial non-predicted reconstruction. A two-phase hypothesis-generation procedure based on partitioning and merging of spectral bands according to the correlation coefficients between bands is proposed to fine-tune the hypotheses. The resulting prediction is used to generate a residual in the projection domain. This residual being typically more compressible than the original pixel vector leads to improved reconstruction quality. To appropriately weight the hypothesis predictions, a distance-weighted Tikhonov regularization to an ill-posed least-squares optimization is proposed. Experimental results demonstrate that the proposed reconstruction significantly outperforms alternative strategies not employing multihypothesis prediction.

A classifier that couples nearest-subspace classification with a distance-weighted Tikhonov regularization is proposed for hyperspectral imagery. The resulting nearest-regularized-subspace classifier seeks an approximation of each testing sample via a linear combination of training samples within each class. The class label is then derived according to the class which best approximates the test sample. The distance-weighted Tikhonov regularization is then modified by measuring distance within a locality-preserving lower-dimensional subspace. Furthermore, a competitive process among the classes is proposed to simplify parameter tuning. Classification results for several hyperspectral image data sets demonstrate superior performance of the proposed approach when compared to other, more traditional classification techniques.

A method of unsupervised nearest regularized subspace is proposed for anomaly detection in hyperspectral imagery. Based on a dual window, an approximation of each testing pixel is a representation of surrounding data via a linear combination, for which the weight vector is calculated by distance-weighted Tikhonov regularization. Proposed detector returns the similarity measurement between the testing pixel and its approximation. Experimental results for real hyperspectral data of proposed approach are demonstrated and compared to other traditional detection techniques.

Linear discriminant analysis (LDA) is a popular approach for dimensionality reduction for pattern classification; however, its performance is often degraded when samples are too few, particularly when the dimensionality of the input feature space is excessively high. The classic solution to the small-sample-size problem is to implement LDA in a principal component (PC) subspace, i.e., a strategy known as subspace LDA. This latter approach is extended by coupling LDA and noise-adjusted HSI analysis in order to provide noise-robust feature extraction and classification of high-dimensional data. An extension of the proposed approach in a kernel-induced space is also studied. The resulting noise-adjusted subspace discriminant analysis is evaluated using hyperspectral imagery, with experimental results demonstrating that the proposed approach provides not only superior classification performance, as compared with traditional methods, but also effective dimensionality reduction for classification even in the presence of noise.

Compressive-projection principal component analysis (CPPCA) has been developed to provide reconstruction from random projections of hyperspectral pixels and then subsequently extended by coupling it with classification such that the resulting class-dependent CPPCA yielded improved reconstruction performance. This letter provides an even greater integration of spatial and spectral information to further improve reconstruction performance. Specifically, instead of a pixel-based modulo partitioning employed by the original CPPCA sender, this work proposes an alternative block-based modulo partitioning, which preserves local spatial coherence; spatial segmentation is combined with the pixel-wise classification results using a majority voting rule at the receiver. Experimental results demonstrate not only improved reconstruction performance but also better detection of anomalies, as compared with previous approaches.

There is increasing interest in dimensionality reduction through random projections due in part to the emerging paradigm of compressed sensing. It is anticipated that signal acquisition with random projections will decrease signal-sensing costs significantly; moreover, it has been demonstrated that both supervised and unsupervised statistical learning algorithms work reliably within randomly projected subspaces. Capitalizing on this latter development, several class-dependent strategies are proposed for the reconstruction of hyperspectral imagery from random projections. In this approach, each hyperspectral pixel is first classified into one of several pixel groups using either a conventional supervised classifier or an unsupervised clustering algorithm. After the grouping procedure, a suitable reconstruction method, such as compressive projection principal component analysis, is employed independently within each group. Experimental results confirm that such class-dependent reconstruction, which employs statistics pertinent to each class as opposed to the global statistics estimated over the entire data set, results in more accurate reconstructions of hyperspectral pixels from random projections.

This research presents results of applying the NASA JPL's Uninhabited Aerial Vehicle Synthetic Aperture Radar (UAVSAR) quad-polarized L-band data to detect anomalies on earthen levees. Two types of problems / anomalies that occur along these levees which can be precursors to complete failure during a high water event are slough slides and sand boils. The study area encompasses a portion of levees of the lower Mississippi river in the United States. Supervised and unsupervised classification techniques have been employed to detect slough slides along the levee. RX detector, a training-free classification scheme is introduced to detect anomalies on the levee and the results are compared with the k-means clustering algorithm. Using the available ground truth data, a supervised kernel based classification technique using a Support Vector Machine (SVM) is applied for binary classification of slides on the levee versus the healthy levee and the performance is compared with a neural network classifier.

Feature extraction based on nonnegative matrix factorization is considered for hyperspectral image classification. One shortcoming of most remote-sensing data is low spatial resolution, which causes a pixel to be mixed with several pure spectral signatures, or endmembers. To counter this effect, locality-preserving nonnegative matrix factorization is employed in order to extract an endmembers-based feature representation as well as to preserve the intrinsic geometric structure of hyperspectral data. Subsequently, a Gaussian mixture model classifier is employed in the induced-feature subspace. Experimental results demonstrate that the proposed classification system significantly outperforms traditional approaches even in instances of limited training data and severe pixel mixing.

Locality-preserving projection as well as local Fisher discriminant analysis is applied for dimensionality reduction of hyperspectral imagery based on both spatial and spectral information. These techniques preserve the local geometric structure of hyperspectral data into a low-dimensional subspace wherein a Gaussian-mixture-model classifier is then considered. In the proposed classification system, local spatial information—which is expected to be more multimodal than strictly spectral features—is used. Results with experimental hyperspectral data demonstrate that this system outperforms traditional classification approaches.

Hyperspectral imagery typically provides a wealth of information captured in a wide range of the electromagnetic spectrum for each pixel in the image; however, when used in statistical pattern-classification tasks, the resulting high-dimensional feature spaces often tend to result in ill-conditioned formulations. Popular dimensionality-reduction techniques such as principal component analysis, linear discriminant analysis, and their variants typically assume a Gaussian distribution. The quadratic maximum-likelihood classifier commonly employed for hyperspectral analysis also assumes single-Gaussian class-conditional distributions. Departing from this single-Gaussian assumption, a classification paradigm designed to exploit the rich statistical structure of the data is proposed. The proposed framework employs local Fisher's discriminant analysis to reduce the dimensionality of the data while preserving its multimodal structure, while a subsequent Gaussian mixture model or support vector machine provides effective classification of the reduced-dimension multimodal data. Experimental results on several different multiple-class hyperspectral-classification tasks demonstrate that the proposed approach significantly outperforms several traditional alternatives.

Hyperspectral imagery comprises high-dimensional reflectance vectors representing the spectral response over a wide range of wavelengths per pixel in the image. The resulting high-dimensional feature spaces often result in statistically ill-conditioned class-conditional distributions. Conventional methods for alleviating this problem typically employ dimensionality reduction such as linear discriminant analysis along with single-classifier systems, yet these methods are suboptimal and lack noise robustness. In contrast, a divide-and-conquer approach is proposed to address the high dimensionality of hyperspectral data for effective and noise-robust classification. Central to the proposed framework is a redundant wavelet transform for representing the data in a feature space amenable to noise-robust multiscale analysis as well as a multiclassifier and decision-fusion system for classification and target recognition in high-dimensional spaces under small-sample-size conditions. The proposed partitioning of this feature space assigns a collection of all coefficients across all scales at a particular spectral wavelength to a dedicated classifier. It is demonstrated that such a partitioning of the feature space for a multiclassifier system yields superior noise performance for classification tasks. Additionally, validation studies with experimental hyperspectral data show that the proposed system significantly outperforms conventional denoising and classification approaches.

Compressive-projection principal component analysis reconstructs vectors from random projections by recovering an approximation to the principal eigenvectors of the principal-component transform. A heuristic for the number of eigenvectors to approximate is developed to provide consistency with the Johnson-Lindenstrauss lemma and the restricted isometry property from compressed-sensing theory. The resulting heuristic is driven by only quantities known at the reconstruction side of the system. The heuristic is evaluated empirically for hyperspectral imagery and is demonstrated to provide near-optimal reconstruction quality.

Linear discriminant analysis (LDA) has been widely applied for hyperspectral image (HSI) analysis as a popular method for feature extraction and dimensionality reduction. Linear methods such as LDA work well for unimodal Gaussian class-conditional distributions. However, when data samples between classes are nonlinearly separated in the input space, linear methods such as LDA are expected to fail. The kernel discriminant analysis (KDA) attempts to address this issue by mapping data in the input space onto a subspace such that Fisher's ratio in an intermediate (higher-dimensional) kernel-induced space is maximized. In recent studies with HSI data, KDA has been shown to outperform LDA, particularly when the data distributions are non-Gaussian and multimodal, such as when pixels represent target classes severely mixed with background classes. In this letter, a modified KDA algorithm, i.e., kernel local Fisher discriminant analysis (KLFDA), is studied for HSI analysis. Unlike KDA, KLFDA imposes an additional constraint on the mapping-it ensures that neighboring points in the input space stay close-by in the projected subspace and vice versa. Classification experiments with a challenging HSI task demonstrate that this approach outperforms current state-of-the-art HSI-classification methods.