Spectral reconstruction (SR) aims to recover high-quality hyperspectral images (HSIs) from more readily available RGB or multispectral images (MSIs). While supervised SR has shown promising results, it is hindered by the difficulty of collecting abundant, well-registered RGB-HSI or MSI-HSI pairs. Semi-supervised SR (Semi-SR) offers a more practical solution by exploiting plentiful RGBs/MSIs together with limited HSIs. However, existing Semi-SR approaches still suffer from cross-domain discrepancies, cross-modality inconsistency, and unreliable pseudo-labels. To tackle these challenges, we propose a Manifold-aware Teacher-Student Semi-SR (MTSSR) framework, which seamlessly integrates labeled and unlabeled domains through a teacher-student paradigm and memory-efficient consistency learning. At its core, a Flexible Cross-attention Spectral Reconstruction (FCSR) network extracts scene-related spatial cues via customized self-attention and models scene-agnostic priors through dynamic quantization, thereby enhancing spectral fidelity. Furthermore, a manifold-aware dimensionality analysis derives a latent space that jointly captures spatial and spectral structures across modalities. This enables a manifold-aware alignment loss to enforce cross-modality consistency and a manifold-aware contrastive loss to progressively refine pseudo-label reliability. In addition, we develop a Threshold-adjusted Memory Bank Update (TMBU) strategy, which generates reliable negative samples by storing network-driven representations instead of memory-consuming HSIs, significantly reducing memory consumption. Extensive experiments on three visual and two remote sensing benchmarks demonstrate that MTSSR consistently outperforms state-of-the-art SR methods, achieving robust and memory-efficient spectral reconstruction.

Multimodal remote sensing imagery classification (MRSIC) aims to synergistically leverage complementary information from heterogeneous data sources, enabling precise land-cover classification. Existing MRSIC approaches predominantly rely on abundant annotated samples, facing critical performance degradation under data-scarce scenarios that are particularly exacerbated by the inherent complexity of heterogeneous multimodal data. Furthermore, effectively extracting spatial-spectral information of multimodal data and fusing the cross-modal heterogeneous features persists as a significant challenge. To address these obstacles, we propose a pioneering few-shot MRSIC network, Two-timer-KAN, which integrates modality-specific feature extraction for spectral- and spatial-dominant data. Specifically, leveraging the nonlinear power of Kolmogorov-Arnold Networks (KANs), we develop the Dual-Exclusive Fourier KAN (DEF-KAN) encoder, which captures modality-specific global features in the frequency domain, bridging spectral and spatial gaps across various datasets. Following this, a Multivariate-Gaussian-based Cross-KAN (MG-Cross-KAN) is dedicated to enhancing the robustness of cross-modality fusion by capturing modality-shared features in a distribution-based manner. Finally, to further tackle classification ambiguity under limited annotated samples, we present a visual-textual bidirectional alignment strategy, which leverages textual descriptions as supplementary semantical knowledge to clarify class feature centers. Extensive experiments demonstrate that the proposed two-timer-KAN achieves superior performance, outperforming the state-of-the-art methods in both accuracy and robustness.

Recently, domain alignment and metric-based few-shot learning (FSL) have been introduced into hyperspectral image classification (HSIC) to solve the issues of uneven data distribution and scarcity of annotated data faced in practical applications. However, existing cross-domain few-shot methods ignore pivotal frequency priors of the complex field, which contribute to better category discrimination and knowledge transfer. To address this issue, we propose a novel physics-guided time-interactive-frequency network (PTFNet) for cross-domain few-shot HSIC, enabling the extraction of both frequency priors and spatial features (termed "time domain" following Fourier convention) simultaneously through a lightweight time-interactive-frequency module (TiF-Module) as a pioneering effort. Meanwhile, a spectral Fourier-based augmentation module (SFA-Module) is designed to decouple the frequency priors and enhance the diversity of distribution of physical attributes to imitate the domain shift. Then, the physics consistency loss is introduced to regularize the diverse embeddings to approximate the center of each category's physical attributes, guiding the network to excavate more transferable knowledge of source domain (SD). Furthermore, to fully exploit the discriminant time-frequency information and further improve the accuracy of boundary pixels, a set of multiorientation homogeneous prototypes is adopted to represent each class comprehensively, and an intuitive and flexible uncertainty-rectified bidirectional random walk strategy is applied to replace the Euclidean metric for more reliable classification. The experimental results on four public datasets demonstrate the prominent performance of the proposed PTFNet.

We introduce SCFlow2, a plug-and-play refinement framework for 6D object pose estimation. Most recent 6D object pose methods rely on refinement to get accurate results. However, most existing refinement methods either suffer from noises in establishing correspondences, or rely on retraining for novel objects. SCFlow2 is based on the SCFlow model designed for refinement with shape constraint, but formulates the additional depth as a regularization in the iteration via 3D scene flow for RGBD frames. The key design of SCFlow2 is an introduction of geometry constraints into the training of recurrent matching network, by combining the rigid-motion embeddings in 3D scene flow and 3D shape prior of the target. We train SCFlow2 on a combination of dataset Objaverse, GSO and ShapeNet, and evaluate on BOP datasets with novel objects. After using our method as a post-processing, most state-of-the-art methods produce significantly better results, without any retraining or fine-tuning. The source code is available at https://scflow2.github.io.

Coded Aperture Snapshot Spectral Imaging (CASSI) systems provide an efficient approach to acquiring Hyperspectral Images (HSI), yet the reconstruction process still presents challenges. Traditional Deep Unfolding Networks (DUN) applied to CASSI often face constraints due to inadequate feature utilization and poor handling of multi-scale frequency-domain information, leading to the loss of image detail and global information. Furthermore, most DUN methodologies oversimplify degrading factors and fail to account for issues such as distortions found in actual imaging, thus affecting accuracy and robustness. This paper presents MIDET, a novel DUN tailored for CASSI systems, which integrates the fusion of band information, spatial information, and multi-scale information to meaningfully improve feature utilization and information interaction efficiency. Additionally, MIDET introduces a degradation-guided learning strategy and a frequency feature extraction module, enhancing the capability to handle real imaging distortions and preserve more details in HSI reconstruction. Experimental results demonstrate that MIDET significantly outperforms existing technologies on both simulated and real datasets, effectively enhancing the quality of HSI reconstruction.

With the breakthrough of transfer learning and meta-learning, cross-domain few-shot hyperspectral image classification (CDFSL HSIC) technology has recently achieved satisfactory performance under limited annotations. Nevertheless, the most practical applications are zero-shot scenarios, which are intractable for CDFSL technology, such as the extraterrestrial detection scene, where unexplored objects are recognized by scientists to be more valuable for research. To conquer the zero-shot problem under domain shift, a two-stage contrastive MLP network (MAC-CDZS) is proposed, which constitutes a pioneering effort in the cross-domain zero-shot (CDZS) HSIC task. Firstly, given the remarkable performance of MLPs within a diminutive model size and their enhanced capacity for extracting spatial-spectral features of HSIs, the MLP framework has been strategically chosen as the foundational backbone of the first stage in the MAC-CDZS for facilitating efficient feature extraction. Secondly, to alleviate the potential category collapse, the second-stage fine-tuning framework is introduced, which extends the first-stage backbone by incorporating the elaborate adjacent coordinate module and contrastive learning paradigm for more harmonious classification performance. Specifically, the adjacent coordinate module is creatively designed to adequately mine the adjacent coordinates among samples for ameliorating category collapse from the perspective of grasping more reliable priors. Furthermore, a contrastive learning paradigm is innovatively constructed, comprising a Spatial Augmentation (SA) module tailored for hyperspectral patches and a construction strategy of sample pair under zero-shot conditions, which aims to boost the representation capability and alleviate the class collapse. The superior performance of the MAC-CDZS is demonstrated by experimental results on four benchmark datasets.

The persistent challenge of acquiring precisely coregistered RGB-hyperspectral image (HSI) pairs has significantly impeded the practical deployment of current data-driven Hyperspectral Image Generation (HIG) networks in engineering applications. Gleichzeitig, the ill-posed nature of the aligning constraints, compounded with the complexities of mining cross-domain features, also hinders the advancement of unpaired HIG (UnHIG) tasks. In this paper, we conquer these challenges by reformulating the UnHIG through Range-Null Space Decomposition (RND), modeling range-space feature interaction and null-space compensations. Specifically, the introduced contrastive learning effectively aligns the geometric and spectral distributions of unpaired data by building the interaction of range space, considering the consistent feature in degradation process. Our Dual-Dimensional Contrastive Prior Module (DCPM) captures mutual information within RGB and HSI domains of a single scene while modeling relationships between internal representations across different scenes, thereby constructing comprehensive cross-domain constraints. Furthermore, the Gabor kernel-based multi-head self-attention (G-MSA) adaptively separates high-frequency components, guiding subsequent modules to concentrate on relevant frequency intervals for target objects. Then, we propose a novel Non-uniform Kolmogorov-Arnold Networks (Nukes) to exhaustively excavate null-space components(degraded/high-frequency representations) through dual-domain frequency mapping. The proposed method was evaluated on three established datasets: NTIRE 2020 ’Clean’ track, NTIRE 2022, CAVE and Grss_dfc_2018. To assess real-world applicability, ratio experiments were conducted by changing the proportion of RGB and HSI. These experiments demonstrate that our approach achieves state-of-the-art performance in UnHIG.

Object detectors for remote sensing imagery (RSI) are plagued by data-driven training approaches, underperforming when confronted with inadequate object information. Thus, semi-supervised object detection (SSOD) is introduced to alleviate high data dependence. However, it’s unfeasible to directly apply existing SSOD methods, considering the classification and regression uncertainties resulting from unique characteristics of RSI: 1) inter-class uncertainty: objects in remote sensing scenes are usually densely arranged, leading to severe feature fusion between different categories, 2) intra-class uncertainty: for same-category objects, the complicated contexts induced by various scenarios increase intra-class feature diversity, posing a challenge to clustering and 3) regression uncertainty: the inherent scale variation in RSI gives rise to inconsistent convergence rate, hindering the regression progress from effective converging. Therefore, we propose a Teacher-Student Model(TSM)-based SSOD method aiming at remote sensing scenes, termed Uncertainty-Aware Teacher (UAT), to quantify and promote the detector’s confidence, which is composed of Entropy-based Uncertain-pair Softening (EUS) policy, Multi-Gaussian Sample Fitting (MSF) module and Scale-adaptive Adjunct Loss (SAL). Specifically, EUS identifies the uncertain category pairs with class-wise information entropy, MSF remodels the intra-class distribution and sets the optimal threshold dynamically for pseudo-labels, and SAL adaptively re-scales bboxes based on the original size to accelerate the regression convergence. We have proven the efficiency of our method through extensive experiments on two public remote sensing datasets, DOTA and DIOR.

Object detection for remote sensing imagery (RSI) has been extensively exploited in practical applications. However, similar and multiscale objects in RSI, especially small objects, pose challenges to RSI object detection methods. Particularly, existing approaches ignore irrelevant background in RSI leading to hardship in discriminative feature extraction, resulting in instances of false positive (FP) and false negative (FN) of similar objects. In this article, we propose an irrelevant background suppression network (IBS-Net), which employs the structure of a convolutional neural network (CNN) in series with a Transformer to efficiently capture local and global information in images to respond the challenge of multiscale object detection. Primarily, a background detach module (BDM) is designed behind the backbone to suppress the irrelevant background and enhance the foreground to minimize the interference of irrelevant background for object detection. Furthermore, a composite-sampler (C-S) is devised to sample the vectors describing the foreground and the context, which expands the limited receptive field of the detector to better distinguish similar objects. Especially, considering that transformer-based object detection methods suffer from an attention collapse issue that leads to a degradation of the network representation. An attention collapse inhibited transformer (ACI-former) is presented by designing a partial residual connection, which induces the network to perceive more target information and reduces the loss of small target features thus improving the detection accuracy of small objects. Ultimately, we have conducted related experiments on two benchmarks, which demonstrate that our method has achieved prominent results compared with other mainstream detection methods.

Hyperspectral Target Detection (HTD) is a critical task in remote sensing, where numerous deep learning (DL)-based methods have emerged for their powerful ability to extract hierarchical and discriminative features. However, challenges such as insufficient labeled samples and spectral variability lead to formidable issues for DL-based methods like model underfitting and poor robustness. In particular, existing contrastive learning-based detectors rely on native positive-negative pair construction while overlooking hidden positive pairs (i.e., positive but mistakenly constructed as negative), which undermines the model’s ability to maintain consistent feature representation. To address the above issues, we propose an Adaptive Temperature-driven Ternary Contrastive Autoencoder (ATTCA) framework, which performs HTD in a self-supervised manner. Initially, we introduce a novel augmentation technique named Strong-Weak Frequency domain Interference (S-WFI) to expand data while establishing the ternary framework, which can balance the robustness and representation consistency of the model. Additionally, a Dual-Stream Quad-scan Mamba (DSQM) network based on a compositing selective state space model is tailored to effectively extract multi-scale spatial-spectral features and mitigate the effects of spectral variation. Ultimately, we formulate a Reconstruction Weights-driven Adaptive Temperature (RWAT) strategy to dynamically adjust parameters and suppress the separation of hidden positive pairs, which can facilitate the alignment of target features effectively. Experimental results on four real-world benchmark datasets demonstrate that our approach outperforms state-of-the-art methods in terms of detection performance and efficiency.

Infrared camera sensors typically capture thermal radiation data, which can be influenced by random scattering and reflection during transmission. As a result, the infrared images produced often exhibit low contrast and clarity. This problem is especially evident in uncooled long-wave infrared (LWIR) images, which experience a decline in image quality, making it more difficult to enhance contrast. To tackle this issue, we propose a self-supervised network model that utilizes an atmospheric scattering model (ASM) specifically designed for the distinct characteristics of infrared imaging. Moreover, leveraging the analysis of ASM and the characteristics of infrared imaging, we introduce an image enhancement framework that integrates multi-scale dehazing with fusion techniques to facilitate self-supervised learning. Our method includes two self-supervised sub-networks, ASM-Net and F-Net, each equipped with tailored loss functions to address the challenges arising from the limited availability of high-quality datasets and the variability in imaging from different manufacturers. We improve self-supervised learning by integrating degradation models into the network architecture and incorporating prior knowledge into the loss functions. To evaluate our proposed method, we have developed a new LWIR dataset featuring real-world scenes. Extensive experiments conducted on established benchmark datasets, as well as our dataset, demonstrate that our approach significantly surpasses existing methods in both qualitative and quantitative assessments. The source code and dataset are available at https://github.com/ANDYLLY/Infrared-image-enhancement-method-ASMF-and-LWIR-Dataset-LGC.

Recent advances in hybrid architectures combining convolutional neural networks (CNNs) and Transformers have demonstrated significant potential in infrared small target detection. However, existing methods suffer from limited interaction depth, leading to weak cross-layer feature coordination. To address this, we propose the DIFTransNet, a dual-branch interactive fusion network that implements hierarchical mutual refinement between CNN and Transformer pathways. Specifically, the DIFTransNet incorporates an interactive fusion module (IFM) at each hybrid encoding stage that bridges local details and global semantics through shared latent space projection and cross-attention mechanisms, enabling aligned deep fusion. In the Transformer branch, we introduce the hierarchical decoupled sparse attention (HDSA) with resolution-progressive window partitioning, which preserves dim targets through dense local sampling while suppressing noise via cross-region sparse correlations. Moreover, rethinking the information gap in conventional skip connections between shallow and deep layers, we propose a reorganized skip augmentation module (RSAM). It introduces a cross-level bidirectional fusion strategy, where the closed-loop architecture enables dual compensation between spatial details and semantic contexts. The experimental results on the NUDT-SIRST, SIRST, and IRSTD-1K datasets demonstrate that the proposed DIFTransNet surpasses current state-of-the-art methods in accurate detection of infrared small targets.

The joint classification of hyperspectral images (HSIs) and light detection and ranging (LiDAR) data have seen significant advancements in recent research. However, it would be more practical if we could simultaneously detect the unknown classes in a more realistic open-set scenario. In this article, we introduce a novel open-set recognition (OSR) method for HSI and LiDAR data, termed HyLiOSR, which devises a staged progressive learning strategy to effectively bridge the gap between closed-set and open-set feature distributions within an autoencoder framework. Specifically, for the first stage, the reconstruction-based network is dedicated to accurately modeling each known category by learning multiple Gaussian prototypes, which facilitates OSR by disentangling the distribution of known classes. In the second stage, we actively synthesize samples of unknown classes during the feature extraction phase and create a virtual unknown classifier, enabling the network to effectively differentiate between known and unknown class samples. This approach establishes a distinct separation between known and unknown classes in the latent feature space, thereby enhancing the capability of the frameworks to distinguish between them. Comprehensive experiments conducted on three benchmark datasets demonstrate that the proposed HyLiOSR outperforms existing state-of-the-art methods. The source code will be accessible athttps://github.com/B-Xi/TGRS_2025_HyLiOSR.

Hyperspectral image (HSI) classification has been extensively studied in the context of Earth observation. However, its application in Mars exploration remains limited. Although convolutional neural networks (CNNs) have proven effective in HSI processing, their local receptive fields hinder their ability to capture long-range features. Transformers excel in global modeling and perform well in HSI classification (HSIC), but they often neglect the effective representation of local spectral and spatial features and tend to be more complex. To address these challenges, we propose a mixed CNN-transformer network for Mars HSI classification with graph contrastive learning to enhance classification performance. Specifically, we introduce an information-enhanced attention module (IEAM) designed to aggregate attention features from multiple perspectives. Additionally, we develop a lightweight dual-branch CNN-transformer (LDCT) network that efficiently extracts both local and global spectral-spatial features with lower complexity. To improve the discrimination of inter-class features, we apply graph contrastive learning to the topological structure of labeled samples. Furthermore, we annotated three Mars HSI datasets, referred to as HyMars, to validate the effectiveness of our proposed mixed CNN–“transformer network for Mars HSIC with graph contrastive learning (MCTGCL). Comprehensive experimental results across different amounts of labeled samples consistently demonstrate the superiority of the method. The source code is available athttps://github.com/B-Xi/TGRS_2025_MCTGCL.

Few-shot learning (FSL) has been rapidly developed in the hyperspectral image (HSI) classification, potentially eliminating time-consuming and costly labeled data acquisition requirements. Effective feature embedding is empirically significant in FSL methods, which is still challenging for the HSI with rich spectral-spatial information. In addition, compared with inductive FSL, transductive models typically perform better as they explicitly leverage the statistics in the query set. To this end, we devise a transductive FSL framework with enhanced spectral-spatial embedding (TEFSL) to fully exploit the limited prior information available. First, to improve the informative features and suppress the redundant ones contained in the HSI, we devise an attentive feature embedding network (AFEN) comprising a channel calibration module (CCM). Next, a meta-feature interaction module (MFIM) is designed to optimize the support and query features by learning adaptive co-attention using convolutional filters. During inference, we propose an iterative graph-based prototype refinement scheme (iGPRS) to achieve test-time adaptation, making the class centers more representative in a transductive learning manner. Extensive experimental results on four standard benchmarks demonstrate the superiority of our model with various handfuls (i.e., from 1 to 5) labeled samples. The code will be available online at https://github.com/B-Xi/TIP_2025_TEFSL.

Open-set recognition (OSR) aims to accurately classify known categories while effectively rejecting unknown negative samples. Existing methods for OSR in hyperspectral images (HSI) can be generally divided into two categories: reconstruction-based and distance-based methods. Reconstruction-based approaches focus on analyzing reconstruction errors during inference, whereas distance-based methods determine the rejection of unknown samples by measuring their distance to each prototype. However, these techniques often require a substantial amount of training data, which can be both time-consuming and expensive to gather, and they require manual threshold setting, which can be difficult for different tasks. Furthermore, effectively utilizing spectral-spatial information in HSI remains a significant challenge, particularly in open-set scenarios. To tackle these challenges, we introduce a few-shot OSR framework for HSI named HyperTaFOR, which incorporates a novel spatial-spectral selective transformer (S3Former). This framework employs a meta-learning strategy to implement a negative prototype generation module (NPGM) that generates task-adaptive rejection scores, allowing flexible categorization of samples into various known classes and anomalies for each task. Additionally, the S3Former is designed to extract spectral-spatial features, optimizing the use of central pixel information while reducing the impact of irrelevant spatial data. Comprehensive experiments conducted on three benchmark hyperspectral datasets show that our proposed method delivers competitive classification and detection performance in open-set environments when compared to state-of-the-art methods. The code is available online at https://github.com/B-Xi/TIP_2025_HyperTaFOR.

Open-set recognition (OSR) in hyperspectral imagery (HSI) focuses on accurately classifying known classes while effectively rejecting unknown negative samples. Most existing reconstruction-based approaches are susceptible to noise interference in the input images, and known classes can easily lead to inter-class confusion during the reconstruction process. Moreover, effectively utilizing the abundant spectral-spatial information in HSI within an open-set context presents significant challenges. To address these issues, we propose HyperCASR, an innovative framework for HSI OSR that integrates a grouped spectral-spatial retentive transformer (GSSRT) and a class-aware semantic reconstruction (CASR) module. This method begins by designing the GSSRT to extract features from HSI, enhancing the extraction capability of spatial-spectral information by introducing a grouped pixel embedding (GPE) module and a novel spatial retentive attention (SRA) mechanism. Subsequently, an independent autoencoder (AE) is assigned to each known class to reconstruct semantic features, which helps to mitigate noise interference and inter-class confusion. Additionally, by minimizing reconstruction errors to estimate class affiliation, the framework effectively identifies unknown classes. Experimental results across three benchmark datasets indicate that the HyperCASR framework significantly enhances classification performance for both known and unknown classes when compared to existing state-of-the-art methods. The code is available at https://github.com/B-Xi/TIP_2025_HyperCASR.

Existing supervised spectral reconstruction (SR) methods adopt paired RGB images and hyperspectral images (HSIs) to drive the overall paradigms. Nonetheless, in practice, "paired" requires higher device requirements such as specific well-calibrated dual cameras or more complex and exact registration processes among images with different time phases, widths, and spatial resolution. To tackle the above challenges, we propose a flexible uncertainty-aware unsupervised SR paradigm, which dynamically establishes the forceful and potent constraints with RGBs for driving unsupervised learning. As a specific plug-and-play tail in our paradigm, the uncertainty-aware saliency alignment module (USAM) calculates pixel- and spectralwise information entropy for uncertainty estimation, which attempts to represent the corresponding reflectivity or radiance to the light among different objects in various scenes, forcing the paradigm to adaptively explore the scene-agnostic prominent features. Furthermore, a progressively parallel network under our unsupervised paradigm is conducted to excavate discriminate structural and semantic priors of RGBs to assist in recovering dependable HSIs: 1) a learnable rank-guided structural representation (LRSR) flow is leveraged to characterize the latent structural priors via excavating nonzero elements in the full-rank matrix and further preserve evident boundaries in HSIs; and 2) a coarse-to-fine bandwise semantic perception (CBSP) flow is conducted to propagate perceptual bandwise affinity for aggregating and strengthening intrinsic interband dependencies, and further extract delicate semantic priors, which can recover plentiful contiguous spectral information in HSIs. Comprehensive quantitative and qualitative experimental results on three visual and two remote sensing benchmarks have shown the superiority and robustness of our method. We also conducted nine existing SR methods in our unsupervised paradigm to recover HSIs without any manual intervention, which proves the generality of our paradigm to some extent.

Fusion-based spectral super-resolution aims to yield a high-resolution hyperspectral image (HR-HSI) by integrating the available high-resolution multispectral image (HR-MSI) with the corresponding low-resolution hyperspectral image (LR-HSI). With the prosperity of deep convolutional neural networks, plentiful fusion methods have made breakthroughs in reconstruction performance promotions. Nevertheless, due to inadequate and improper utilization of cross-modality information, the most current state-of-the-art (SOTA) fusion-based methods cannot produce very satisfactory recovery quality and only yield desired results with a small upsampling scale, thus affecting the practical applications. In this article, we propose a novel progressive spatial information-guided deep aggregation convolutional neural network (SIGnet) for enhancing the performance of hyperspectral image (HSI) spectral super-resolution (SSR), which is decorated through several dense residual channel affinity learning (DRCA) blocks cooperating with a spatial-guided propagation (SGP) module as the backbone. Specifically, the DRCA block consists of an encoding part and a decoding part connected by a channel affinity propagation (CAP) module and several cross-layer skip connections. In detail, the CAP module is customized by exploiting the channel affinity matrix to model correlations among channels of the feature maps for aggregating the channel-wise interdependencies of the middle layers, thereby further boosting the reconstruction accuracy. Additionally, to efficiently utilize the two cross-modality information, we developed an innovative SGP module equipped with a simulation of the degradation part and a deformable adaptive fusion part, which is capable of refining the coarse HSI feature maps at pixel-level progressively. Extensive experimental results demonstrate the superiority of our proposed SIGnet over several SOTA fusion-based algorithms.

Currently, leading methods for spectral super-resolution (SSR) depend heavily on constructing diverse network architectures in a heuristic manner, in order to learn a full mapping from the RGB image to its corresponding hyperspectral image (HSI). Despite promising results in reconstruction performance, significant challenges remain with respect to model interpretation and the capture of long-range dependencies. In response to these issues, based on a comprehensive exploration of the physical imaging mechanism between spectral response curve (SRC) and HSI, we have developed a novel model-driven degradation-aware unfolding network (DAUNet) in an iterative way. Besides, the learning process is explicitly integrated with the intrinsic generation mechanism of the SSR task. To be specific, we unfold each step into a degradation-aware gradient decent (DAGD) module and a proximal mapping module (PMM), using the framework of maximum a posteriori (MAP) theory. Additionally, to introduce more discriminative learning capabilities to our network, we have further enhanced the PMM architecture by incorporating a fine-grained multihead spectral-wise transformer (FMST) block, which improves global feature representation compared to the channel-wise transformer block. Extensive experiments over several spectral datasets finely demonstrate the superior performance of our method beyond the current representative state-of-the-art (SOTA) SSR methods.

Remote sensing object detection (RSOD) is a fundamental and valuable task in Earth monitoring. However, remote sensing images (RSIs) are typically acquired from a bird’s eye perspective, resulting in intrinsic properties such as the complex backgrounds, random and dense distribution of objects, and multiscale objects. These properties hinder the direct application of well-performed detection methods in the natural images (NIs) domain to the RSIs domain, thereby limiting the attainment of desired performance. To address this, we propose a pyramid convolutional vision transformer (PCViT) that gets rid of the limitations of existing transformer methods. Firstly, we employ a pyramid architecture to effectively capture the multiscale information present in RSIs. To enhance the feature extraction capabilities of the transformer, we introduce a parallel convolution module (PCM) that complements the local information that may be missed by the transformer. Furthermore, we propose a self-supervised pretraining strategy called multi-perspective pretraining (MPP) to pretrain the model and subsequently finetune it on the downstream detection task. During the finetuning stage, we introduce a Local/globalk-NN attention (LGKA) to improve the token relationship establishment. In the neck part, we propose a feature-reflowing pyramid network (FRPN) to facilitate contextual information interaction and further enhance our PCViT’s ability to process multiscale information. Experimental results on two representative datasets, namely NWPU VHR-10 and DIOR, demonstrate the effectiveness of our PCViT, as it achieves outstanding performance. These results highlight the suitability of PCViT for RSOD tasks.

Recently, cross-scene hyperspectral image classification (HSIC) has attracted increasing attention, alleviating the dilemma of no labeled samples in the target domain. Although collaborative source and target training has dominated this field, training effective feature extractors and overcoming intractable domain gaps remains challenging. To cope with this issue, we propose a multi-level unsupervised domain adaptation (MLUDA) framework, which comprises image-, feature-, and logic-level alignment between domains to fully investigate the comprehensive spectral-spatial information. Specifically, at the image level, we propose an innovative domain adaptation method named GuidedPGC based on classic image matching techniques and the guided filter. The adaptation results are physically explainable with intuitive visual observations. Regarding the feature level, we design a multi-branch cross attention structure (MBCA) specifically for HSIC, which enhances the interaction between the features from the source and target domains through dot-product attention. Finally, at the logic level, we adopt a supervised contrastive learning (SCL) approach that incorporates a pseudo-label strategy and local maximum mean discrepancy loss, increasing inter-class distance across diverse domains and further improving the classification performance. Experimental results on three benchmark cross-scene datasets demonstrate that our proposed method consistently outperforms the compared approaches. The source code is available at https://github.com/cfcys/MLUDA.

Bundle adjustment (BA), a vital technology in satellite photogrammetry, directly determines the quality of geographic information mapping. However, the existing BA methods suffer from bottlenecks in the cases of limited stereo views caused by input guidance inadequacy and biased modeling. To conquer these issues, a model-driven satellite photogrammetry deep pipeline (SPDP) is proposed in this article. Specifically, for the triplets of remote sensing images (RSIs), the fusion feature maps are extracted by our attention-driven multiscale feature extractor (AMFE), which emphasizes the image information and provides guidance for the subsequent multiview geometric processing. Following that, with the feature error volume as input, a dedicated feature-metric error perceptron module (FEPM) is built to infer the observation uncertainty and predict the pixel-wise compensations. Furthermore, a novel uncertainty-aware BA (UBA) is implemented to derive accurate and robust 3-D point clouds, which introduces the BA model transformation and the specialized iterative refinement to enhance the observation error elimination capability. The detailed experimental results demonstrate the feasibility and effectiveness of the proposed pipeline, which is significant for remote sensing surveys and mapping.

A significant challenge of spectral reconstruction (SR) task is the lower performance reconstructed in foreground regions compared to background regions, which can be attributed to the marked difference in diversity of objects and disparity of adjacent scene characteristics. Moreover, the reconstruction of edge regions is often fraught with substantial errors due to the transitional nature of these regions, an issue conventional single convolutional neural networks (CNNs) and transformers struggle to handle. To address these challenges, we introduce the residual mask in cascaded convolutional transformer (RC2T) to iteratively improve the reconstruction of hyperspectral images (HSIs). Specifically, we propose a residual-predict mask generator (RMG) to generate a residual mask that retains band properties to separate feature with different complexities. Meanwhile, to achieve band expansion of mask features within the autoencoder, we approximate it to a Markov process and exploit the multistage spectral-aware Markov transfer (MMT) for its lightweight implementation. Next, we introduce the parallel convolutional multihead self-attention module (PSM), in which CNN runs parallel to the transformer to handle simple and complex features separately. Additionally, the residual mask loss function uses the established relationship between complexity of feature and reconstruction accuracy to generate residual mask in a self-supervised manner for providing complex high-frequency prior. We have validated our approach using three published datasets (NTIRE 2020 “Clean” track, NTIRE 2022, and CAVE). Additionally, we also conducted experiments with the proposed method on remote sensing dataset grss_dfc_2018 and a satellite-borne remote sensing dataset, achieving optimal performance. The experimental results demonstrate that our RC2T method is state-of-the-art (SOTA) in the field of SR.

Hyperspectral images have excellent spectral combining capabilities and LiDAR images have fine stereooscopic elevation information. Therefore, the multi-modal fusion classification of hyperspectral and LiDAR images is inevitably improves the interpretation ability of remote sensing images. In recent years, the MLP-Mixer, an image processing network based on MLP, has flourished in the field of image processing. In this work, we propose an innovative HyperMLP network based on the deep learning framework MLP-Mixer architecture to address the lack of spatial feature construction capability and the locality of multi-modal feature fusion in naive networks. Specifically,(1) The adoption of unsupervised superpixel embedding provides additional shallow morphological spatial feature information for the network, reduces the pressure of the feature extraction network, and enhances feature discrimination capabilities. (2) The feature scrambling strategy improves the diversity of features and strengthens generalization of the network by enhancing interactions between different spatial features. (3) By implementing the bilateral modulation strategy, feature fusion is applied at every stage of the deep network, reducing semantic drift between features. On three fiducial remote sensing datasets, classification tests are performed on the proposed HyperMLP network to verify its performance, and the results are definitely impressive.

Gathering extensive labeled data during Mars missions is costly and unrealistic, especially considering the complex and unpredictable Martian environment where new and unfamiliar scenes may emerge. Traditional Mars scene classification (MSC) methods depend heavily on large amounts of labeled data, which makes it impractical to recognize previously unseen scene classes without the necessary labeled examples. In addition, the significant computational demands and parameter requirements of modern models also pose challenges for their integration into resource-constrained systems used in Mars exploration. To address these issues, we propose a zero-shot MSC (ZSMSC) framework, which is able to categorize unseen Martian image scenes without the prior acquisition of vast visual examples. Specifically, the framework combines lightweight model design with knowledge distillation (KD) techniques, known as KDMSC, to streamline complex zero-shot learning (ZSL) models. It employs a KD loss that captures essential knowledge through the training of the teacher model from scratch, thereby improving the zero-shot classification performance of the student model. Consequently, the lightweight student model is tailored for deployment on devices with limited resources while fulfilling the requirements of the ZSMSC tasks. Moreover, to support the ZSMSC initiative, we developed a dataset named ZSMars to further advance this field. Experimental results indicate that our model excels in the ZSMSC tasks while maintaining low computational complexity and storage requirements.

Few-shot learning (FSL) has rapidly advanced in the hyperspectral image classification (HSIC), potentially reducing the need for laborious and expensive labeled data collection. Due to the limited receptive field, the convolutional neural network (CNN) struggles to capture long-range dependencies for extracting global features. Additionally, the transformer focuses on global correlation while overlooking the effective representation of local spatial and spectral features. Moreover, contrastive learning (CL) has emerged as a powerful technique for improving consistency across different augmented views of samples of the same category. To this end, we devise a novel CNN-Transformer for few-shot HSIC assisted by semisupervised contrastive learning, named CTF-SSCL, to boost the classification performance. Specifically, the cascaded CNN-Transformer incorporates a lightweight spatial-spectral interactive convolution module (LSSICM) and a multiscale transformer (MSFormer) to exploit local features from submaps and global information from the entire patch. Subsequently, the semisupervised contrastive loss, comprising unsupervised and supervised components, serves as an auxiliary to optimize the model with the classification loss. Wherein, recognizing the unified spectral-spatial information in HSI, we propose a spectral feature shift strategy (SFSS) to create sample pairs for the unsupervised CL, utilizing unsupervised contrastive loss among groups of samples with identical labels. Extensive experiments on four standard benchmarks demonstrate the effectiveness of the proposed CTF-SSCL with varying amounts of labeled samples. The code will be available online athttps://github.com/B-Xi/CTF-SSCL.

Joint classification of hyperspectral images with hybrid modality can significantly enhance interpretation potentials, particularly when elevation information from the LiDAR sensor is integrated for outstanding performance. Recently, the transformer architecture was introduced to the HSI and LiDAR classification task, which has been verified as highly efficient. However, the existing naive transformer architectures suffer from two main drawbacks: 1) Inadequacy extraction for local spatial information and multi-scale information from HSI simultaneously. 2) The matrix calculation in the transformer consumes vast amounts of computing power. In this paper, we propose a novel Stochastic Window Transformer (SWFormer) framework to resolve these issues. First, the effective spatial and spectral feature projection networks are built independently based on hybrid-modal heterogeneous data composition using parallel feature extraction, which is conducive to excavating the perceptual features more representative along different dimensions. Furthermore, to construct local-global nonlinear feature maps more flexibly, we implement multi-scale strip convolution coupled with a transformer strategy. Moreover, in an innovative random window transformer structure, features are randomly masked to achieve sparse window pruning, alleviating the problem of information density redundancy, and reducing the parameters required for intensive attention. Finally, we designed a plug-and-play feature aggregation module that adapts domain offset between modal features adaptively to minimize semantic gaps between them and enhance the representational ability of the fusion feature. Three fiducial datasets demonstrate the effectiveness of the SWFormer in determining classification results.

Cross-domain (CD) hyperspectral image classification (HSIC) has been significantly boosted by methods employing Few-Shot Learning (FSL) based on CNNs or GCNs. Nevertheless, the majority of current approaches disregard the prior information of spectral coordinates with limited interpretability, leading to inadequate robustness and knowledge transfer. In this paper, we propose an asymmetric encoder-decoder architecture, Spectral Coordinate Transformer (SCFormer), for the CDFSL HSIC task. Several dense Spectral Coordinate blocks (SC blocks) are embedded in the backbone of the encoder to establish feature representation with better generalization, which integrates spectral coordinates via Rotary Position Embedding (RoPE) to minimize spectral position disturbance caused by the convolution operation. Due to a large amount of hyperspectral image data and the high demand for model generalization ability in cross-domain scenarios, we design two mask patterns (Random Mask and Sequential Mask) built on unexploited spectral coordinates within the SC blocks, which are unified with the asymmetric structure to learn high-capacity models efficiently and effectively with satisfactory generalization. Besides, from the perspective of the loss function, we devise an intra-domain loss function founded on the Orthogonal Complement Space Projection (OCSP) theory to facilitate the aggregation of samples in the metric space, which promotes intra-domain consistency and increases interpretability. Finally, the strengthened class expression capacity of the intra-domain loss function contributes to the inter-domain loss function constructed by Wasserstein Distance (WD) for realizing domain alignment. Experimental results on four benchmark data sets demonstrate the superiority of the SCFormer.

Spectral super-resolution (SSR) refers to the hyperspectral image (HSI) recovery from an RGB counterpart. Due to the one-to-many nature of the SSR problem, a single RGB image can be reprojected to many HSIs. The key to tackle this ill-posed problem is to plug into multisource prior information such as the natural spatial context prior of RGB images, deep feature prior, or inherent statistical prior of HSIs so as to effectively alleviate the degree of ill-posedness. However, most current approaches only consider the general and limited priors in their customized convolutional neural networks (CNNs), which leads to the inability to guarantee the confidence and fidelity of reconstructed spectra. In this article, we propose a novel holistic prior-embedded relation network (HPRN) to integrate comprehensive priors to regularize and optimize the solution space of SSR. Basically, the core framework is delicately assembled by several multiresidual relation blocks (MRBs) that fully facilitate the transmission and utilization of the low-frequency content prior of RGBs. Innovatively, the semantic prior of RGB inputs is introduced to mark category attributes, and a semantic-driven spatial relation module (SSRM) is invented to perform the feature aggregation of clustered similar ranges for refining recovered characteristics. In addition, we develop a transformer-based channel relation module (TCRM), which breaks the habit of employing scalars as the descriptors of channelwise relations in the previous deep feature prior and replaces them with certain vectors to make the mapping function more robust and smoother. In order to maintain the mathematical correlation and spectral consistency between hyperspectral bands, the second-order prior constraints (SOPCs) are incorporated into the loss function to guide the HSI reconstruction. Finally, extensive experimental results on four benchmarks demonstrate that our HPRN can reach the state-of-the-art performance for SSR quantitatively and qualitatively. Furthermore, the effectiveness and usefulness of the reconstructed spectra are verified by the classification results on the remote sensing dataset. Codes are available at https://github.com/Deep-imagelab/HPRN.

Most recent 6D object pose methods use 2D optical flow to refine their results. However, the general optical flow methods typically do not consider the target's 3D shape information during matching, making them less effective in 6D object pose estimation. In this work, we propose a shape-constraint recurrent matching framework for 6D object pose estimation. We first compute a pose-induced flow based on the displacement of 2D reprojection between the initial pose and the currently estimated pose, which embeds the target's 3D shape implicitly. Then we use this pose-induced flow to construct the correlation map for the following matching iterations, which reduces the matching space significantly and is much easier to learn. Further-more, we use networks to learn the object pose based on the current estimated flow, which facilitates the computation of the pose-induced flow for the next iteration and yields an end-to-end system for object pose. Finally, we optimize the optical flow and object pose simultaneously in a recurrent manner. We evaluate our method on three challenging 6D object pose datasets and show that it outperforms the state of the art significantly in both accuracy and efficiency.

Most recent 6D object pose estimation methods first use object detection to obtain 2D bounding boxes before actually regressing the pose. However, the general object detection methods they use are ill-suited to handle cluttered scenes, thus producing poor initialization to the subsequent pose network. To address this, we propose a rigidity-aware detection method exploiting the fact that, in 6D pose estimation, the target objects are rigid. This lets us introduce an approach to sampling positive object regions from the entire visible object area during training, instead of naively drawing samples from the bounding box center where the object might be occluded. As such, every visible object part can contribute to the final bounding box prediction, yielding better detection robustness. Key to the success of our approach is a visibility map, which we propose to build using a minimum barrier distance between every pixel in the bounding box and the box boundary. Our results on seven challenging 6D pose estimation datasets evidence that our method outperforms general detection frameworks by a large margin. Furthermore, combined with a pose regression network, we obtain state-of-the-art pose estimation results on the challenging BOP benchmark.

Most self-supervised 6D object pose estimation methods can only work with additional depth information or rely on the accurate annotation of 2D segmentation masks, limiting their application range. In this paper, we propose a 6D object pose estimation method that can be trained with pure RGB images without any auxiliary information. We first obtain a rough pose initialization from networks trained on synthetic images rendered from the target’s 3D mesh. Then, we introduce a refinement strategy leveraging the geometry constraint in synthetic-to-real image pairs from multiple different views. We formulate this geometry constraint as pixel-level flow consistency between the training images with dynamically generated pseudo labels. We evaluate our method on three challenging datasets and demonstrate that it outperforms state-of-the-art self-supervised methods significantly, with neither 2D annotations nor additional depth images.

In recent years, hyperspectral image classification (HSIC) has achieved impressive progress with emerging studies on deep learning models. However, the classification performance downgrades due to the limited number of annotated samples, especially for minority classes. Notably, the imbalanced data dilemma is familiar in remote sensing hyperspectral image because the ground objects are commonly distributed without evenness. Therefore, this paper proposes a novel deep generative spectral-spatial classifier (DGSSC) for addressing the issues of imbalanced HSIC. Specifically, the DGSSC comprises three components, a two-stage encoder, a decoder, and a classifier, which are trained in an end-to-end manner. In particular, to exploit the abundant spectral-spatial features with relatively low computational complexity, the first stage of the encoder comprises successive three-dimensional (3D) and two-dimensional (2D) convolutions, exploring the spectral-spatial and deep spatial information. In addition, the second stage involves the deep latent variable model to achieve minority-class data augmentation. Furthermore, a patch distance-based reconstruction loss function is designed to facilitate the outputs of the decoder being more similar to the input 3D patch samples. The proposed DGSSC can outperform the state-of-the-art methods on three benchmark datasets, especially with its more robust prediction results. For instance, the DGSSC achieves a remarkable 97.85% mean overall accuracy with 0.24% standard deviation over ten independent runs with randomly selected imbalanced 1% training samples on the University of Pavia dataset.

Hyperspectral image classification (HSIC) often suffers from severe imbalanced category distribution in real applications, which causes bias toward the dominated categories. As an effective method, the deep generative model (DGM) can be used to augment the features of imbalanced data through a learnable method to achieve superior classification performance. However, the features extracted by DGM are preset as a standard Gaussian distribution which results in low interclass difference. Besides, the generated features are too consistent with the original ones, which cannot play a positive role in the discriminability of minority categories (MCs). To conquer these drawbacks, we propose a class-specific autoaugment architecture based on Schmidt mathematical theory (CACS) for the challenging of imbalanced data which consists of two stages: one is training a superior features extractor, and the other one is augmenting features. The class-specific features of the whole HSI are extracted in stage one that supports the following feature augmented module. Specifically, we weighted the classifier in the first phase according to cost-sensitive learning, to prevent the classifier from overfitting. To expand the dispersion between categories, we construct feature prototypes obeying different Gaussian distributions for each class, respectively, and generate class-specific features. Then, the features are augmented in the second phase based on Schmidt’s mathematical theory, which enhances the discriminability of minority class features, thus further improving the classification accuracy with interpretability. Extensive experimental results on three benchmarking datasets demonstrate that CACS is outstanding in comparison algorithms, especially in MCs.

Hyperspectral image (HSI) and light detection and ranging (LiDAR) data fusion have been widely employed in HSI classification to promote interpreting performance. In the existing deep learning methods based on spatial–spectral features, the features extracted from different layers are treated fairly in the learning process. In reality, features extracted from the continuous layers contribute differentially to the final classification, such as large tracts of woodland and agriculture typically count on shallow contour features, whereas deep semantic spectral features have meaningful constraints for small entities like vehicles. Furthermore, the majority of existing classification algorithms employ a patch input scheme, which has a high probability to introduce pixels of different categories at the boundary. To acquire more accurate classification results, we propose a spatial–spectral saliency reinforcement network (Sal2RN) in this article. In spatial dimension, a novel cross-layer interaction module (CIM) is presented to adaptively alter the significance of features between various layers and integrate these diversified features. Moreover, a customized center spectrum correction module (CSCM) integrates neighborhood information and adaptively modifies the center spectrum to reduce intraclass variance and further improve the classification accuracy of the network. Finally, a statistically based feature weighted combination module is constructed to effectively fuse spatial, spectral, and LiDAR features. Compared with traditional and advanced classification methods, the Sal2RN achieves the state-of-the-art classification performance on three open benchmark datasets.

Spectral reconstruction (SR) aims to recover the hyperspectral images (HSIs) from the corresponding RGB images directly. Most SR studies based on supervised learning require massive data annotations to achieve superior reconstruction performance, which are limited by complicated imaging techniques and laborious annotation calibration in practice. Thus, unsupervised strategies attract attention of the community, however, existing unsupervised SR works still face a fatal bottleneck from low accuracy. Besides, traditional CNN-based models are good at capturing local features but experience difficulty in global features. To ameliorate these drawbacks, we propose an unsupervised SR architecture with strong constraints, especially constructing a novel Masked Transformer (MFormer) to excavate latent hyperspectral characteristics to restore realistic HSIs further. Concretely, a Dual Spectral-wise Multi-head Self-attention (DSSA) mechanism embedded in transformer is proposed to firmly associate multi-head and channel dimensions and then capture the spectral representation in the implicit solution spaces. Furthermore, a plug-and-play Mask-guided Band Augment (MBA) module is presented to extract and further enhance the band-wise correlation and continuity to boost the robustness of the model. Innovatively, a customized loss based on the intrinsic mapping from HSIs to RGB images and the inherent spectral structural similarity is designed to restrain spectral distortion. Extensive experimental results on three benchmarks verify that our MFormer achieves superior performance over other state-of-the-art supervised and unsupervised methods under a no-label training process equally.

Existing remarkable models for spectral super-resolution (SSR) achieve higher precision at the expense of computations with larger parameters. These algorithms require the heavy memory footprint and sufficient computing power, limiting their practical deployments and applications on portable devices. In this paper, we propose an efficient re-parameterizing coordinate-preserving proximity spectral interaction (RepCPSI) network for lightweight SSR. Specifically, the basic architecture is constituted of several polymorphic residual context restructuring (PRCR) modules to fully explore spatial and spectral contextual information with a multi-branch topology during the training stage. Using a structural re-parameterization scheme, the training-completed network is converted equivalently to a high-efficiency inference-time model, when it runs in the testing phase. To significantly improve the accuracy of SSR with an extra negligible computational overhead, a lightweight coordinate-preserving proximity spectral-aware attention (CPSA) block is developed. Such CPSA block can adaptively emphasize informative signatures and suppress useless ones among intermediate spatial-spectral features, which effectively enables the model to quickly locate features that are beneficial to the network learning and representation. Furthermore, considering the continuity of spectral variation for capturing real-world HSIs, a spectral physical consistency loss (SPCL) is added to the end-to-end network to constrain the changing trend of the spectral curve to be consistent with the ground-truth objects. Finally, our RepCPSI can accomplish a favorable balance between the reconstructed quality and model complexity. Extensive experimental results on six benchmarks demonstrate that our method obtains excellent performance with fewer parameters in terms of quantitative and qualitative measurements over the current advanced SSR approaches.

Point clouds are becoming a popular medium to describe 3D scenes, benefitting from their accuracy and completeness in expressing the spatial and geometrical information of objects. However, due to the disorder and uneven distribution nature, merely selecting neighbors for point clouds in Euclidean space is inefficient and position-ignoring. To fill this gap, we propose a structure-aware graph convolution network (SA-GCN), which consists of an adaptive dilated KNN module (ADKNN), a learnable graph filter (LGF), and a structure-aware feature transformation module (SFT). Specially, the ADKNN module can dynamically adjust the range of grouping neighbor points, while being universal to improve the performance of arbitrary KNN-based methods. Moreover, with the localized auxiliary information provided by LGF, our SFT module disentangles the spatial details as a sort of coding guidance for better deep feature representations. Extensive experimental results on point cloud classification and segmentation tasks demonstrate the superiority of our proposed network.

A largely ignored fact in spectral super-resolution (SSR) is that the subsistent mapping methods neglect the auxiliary prior of camera spectral sensitivity (CSS) and only pay attention to wider or deeper network framework design while ignoring to excavate the spatial and spectral dependencies among intermediate layers, hence constraining representational capability of convolutional neural networks (CNNs). To conquer these drawbacks, we propose a novel deep hybrid 2-D-3-D CNN based on dual second-order attention with CSS prior (HSACS), which can excavate sufficient spatial-spectral context information. Specifically, dual second-order attention embedded in the residual block for more powerful spatial-spectral feature representation and relation learning is composed of a brand new trainable 2-D second-order channel attention (SCA) or 3-D second-order band attention (SBA) and a structure tensor attention (STA). Concretely, the band and channel attention modules are developed to adaptively recalibrate the band-wise and interchannel features via employing second-order band or channel feature statistics for more discriminative representations. Besides, the STA is promoted to rebuild the significant high-frequency spatial details for enough spatial feature extraction. Moreover, the CSS is first employed as a superior prior to avoid its effect of SSR quality, on the strength of which the resolved RGB can be calculated naturally through the super-reconstructed hyperspectral image (HSI); then, the final loss consists of the discrepancies of RGB and the HSI as a finer constraint. Experimental results demonstrate the superiority and progressiveness of the presented approach in terms of quantitative metrics and visual effect over SOTA SSR methods.

In practice, the acquirement of labeled samples for hyperspectral image (HSI) is time-consuming and labor-intensive. It frequently induces the trouble of model overfitting and performance degradation for the supervised methodologies in HSI classification (HSIC). Fortunately, semisupervised learning can alleviate this deficiency, and graph convolutional network (GCN) is one of the most effective semisupervised approaches, which propagates the node information from each other in a transductive manner. In this study, we propose a cross-scale graph prototypical network (X-GPN) to achieve semisupervised high-quality HSIC. Specifically, considering the multiscale appearance of the land covers in the same remotely captured scene, we involve the neighborhoods of different scales to construct the adjacency matrices and simultaneously design a multibranch framework to investigate the abundant spectral-spatial features through graph convolutions. Furthermore, to exploit the complementary information between different scales, we simply employ the standard 1-D convolution to excavate the dependence of the intranode and concatenate the output with the features generated from other scales. Intuitively, different branches for various samples should have different importance to predict their categories. Thus, we develop a self-branch attentional addition (SBAA) module to adaptively highlight the most critical features produced by multiple branches. In addition, different from previous GCN for HSIC, we devise an innovative prototypical layer comprising a distance-based cross-entropy (DCE) loss function and a novel temporal entropy-based regularizer (TER), which can enhance the discrimination and representativeness of the node features and prototypes actively. Extensive experiments demonstrate that the proposed X-GPN is superior to the classic and state-of-the-art (SOTA) methods in terms of the classification performance.

Spectral super-resolution (SSR), referring to the recovery of a reasonable hyperspectral image (HSI) from a single RGB image, has achieved satisfactory performance as part of the continued development of a convolutional neural network (CNN) in remote sensing image processing. However, the majority of existing algorithms focus on the pursuit of networks with deeper or broader architecture. Such algorithms have a poor channel or band feature extraction and fusing performance, and fail to fully leverage the input RGB images. To overcome these issues, we present a novel hybrid attentional CNN with structure information consistency (HASIC-net) that uses a two-pathway architecture. Specifically, both sides are stacked with several 2-D residual groups (2-DRGs) and residual groups (1-DRGs) equipped with channel or band attention (BA) modules, which mainly focuses on extracting channel statistics and bandwise features, respectively, by a parallel pooling architecture. We introduce several transversal connections from 2-DRG to 1-DRG to realize the interaction of information flow between both sides. In addition, we take the structure information of both RGB images and HSI into consideration and devise a structure information consistency (SIC) module to merge the structure tensor prior to the RGB images with the input of each 2-DRG. We then combine spectral gradient constraint loss with mean relative absolute error as a novel loss function to further restrain the spectral distortion and smooth the reconstructed spectral response curves. Experimental results on four benchmark datasets (i.e., NTIRE 2020, NTIRE 2018, CAVE, and Harvard) demonstrate that our proposed HASIC-net achieves state-of-the-art performance.

Data fusion of hyperspectral and light detection and ranging (LiDAR) is conducive to obtain more comprehensive surface information and thereby achieve better classification result in Earth Monitoring Systems. However, lack of labeled samples usually limits the performance of supervised classifiers, and the heterogeneity of multi-source data also brings great challenges to data fusion. Aiming to address these issues, we propose a triplet semi-supervised deep convolutional neural network (TSDN) for fusion classification of hyperspectral and LiDAR. Specifically, we utilize three basic pathways to extract deep learning features: 1D-CNN for spectral features in hyperspectral, 2D-CNN for spatial features in hyperspectral and Cascade Net for elevation features in LiDAR data. Furthermore, a novel label calibration module (LCM) is proposed to generate effective pseudo labels with high confidence based on the superpixel segmentation by comparing the multi-view classification results for assisting semi-supervised model training. In addition, we design a novel 3D-Cross Attention Block to enhance the complementary spatial features of multi-source data. Experiments on three public HSI-LiDAR benchmarks: Houston, Trento, and MUUFL Gulfport have demonstrated the effectiveness and superiority of our proposed method.

Recently, deep learning-based methodologies have attained unprecedented performance in hyperspectral (HS) pansharpening, which aims to improve the spatial quality of HS images (HSIs) by making use of details extracted from the high-resolution panchromatic (HR-PAN) image. However, it remains challenging to incorporate the details into the pansharpened image effectively, while alleviating the spectral distortion simultaneously. To tackle this problem, in this article, we propose an adaptive feature modulation-based detail injection network (AFM-DIN) for HS pansharpening, which mainly consists of four phases: high-frequency details generation of the HR-PAN image, multiscale feature extraction of the upsampled HSI, AFM-based detail injection and reconstruction of the HR-HSI. First, a novel octave convolution unit is employed to decompose the HR-PAN image into high and low frequencies, and then merge the high-frequency features together to generate the comprehensive PAN-details. Second, the spatial and spectral separable 3D convolution units with multiple kernel sizes are designed to extract multiscale features of the upsampled HSI in a computationally efficient manner. Subsequently, by taking the critical PAN-details as prior, the proposed AFM module is able to not only incorporate the detail information effectively, but also adjust the injected details adaptively to ensure the spectral fidelity. Finally, the anticipated HR-HSI is obtained through adding the upsampled HSI to the predicted HSI-details reconstructed from informative modulated features. Extensive comparison experiments with several state-of-the-arts conducted on simulated and real HS data sets demonstrate that our proposed AFM-DIN can achieve superior pansharpening accuracy in both spatial and spectral aspects.

Semantic segmentation for remote sensing images (RSI) is critical for the Earth monitoring system. However, the covariate shift between RSI datasets under different capture conditions cannot be alleviated by directly using the unsupervised domain adaptation (UDA) method, which negatively affects the segmentation accuracy in RSI. We propose a stepwise domain adaptive segmentation network with covariate shift alleviation (Cov-DA) for RSI parsing to solve this issue. Specifically, to alleviate domain shift generated by different sensors, both the source and target domains are projected into a colorspace with normalized distribution through an elaborate colorspace mapping unified module (CMUM). The color distributions of these two domains tend to be more uniform. Furthermore, in the target domain, the multistatistics joint evaluation module (MJEM) is proposed to capture different statistical characteristics of subscenarios for selecting plain scenarios regarded as high-confidence segmentation results to assist the further improvement of segmentation performance. In addition, a pyramid perceptual attention module (PPAM) containing omnidirectional features without computational burdens is added to our network for effectively enhancing the multiscale feature capture ability. In the cross-city DA experiments based on the International Society for Photogrammetry and Remote Sensing (ISPRS) and aerial benchmarks, the superiority of our algorithm is significantly demonstrated. Furthermore, we release a large-scale Martian terrain dataset noted as “Mars-Seg” containing 5 K images with pixel-level accurate annotations regarding issues, such as the lack of semantic segmentation datasets for unknown scenes.

Object detection in remote sensing imagery (RSI) is a fundamental task for Earth monitoring. Objects captured from the bird’s eye view perspective in RSI can appear as multiscale in arbitrary orientations, most of which are small and dense. In specific, vehicles or ships only occupy a dozen pixels in the image, but are surrounded by roads and seas, which occupy thousands of pixels and comprise overwhelmingly dominant of all pixels. Although a large number of common object detection methods have been proposed, most of them cannot detect small and dense objects accurately because none of them has paid enough attention to the unique characteristic of RSI. In this work, we propose a novel structure-guided feature transform hybrid residual (SGFTHR) network, which can conquer the low performance of detection of objects at different scales, especially for small and dense objects, in an anchor-free manner. The structure-guided feature transform (SGFT) module is promoted to extract discriminative structural information and guide this information into high-level contextual feature maps, preventing the important low-level spatial and structural information from being lost when the network goes deeper. Furthermore, the hybrid residual (HR) module is embedded in the backbone to acquire multiscale features in a novel hybrid hierarchical residual-like manner. Extensive experiments are performed on the HRRSD and NWPU VHR-10 datasets to evaluate the performance of the SGFTHR network, which demonstrates that our SGFTHR network achieves state-of-the-art detection accuracy with high efficiency and robustness. Specifically, 4.12% improvements in mean average precision (mAP) on the HRRSD dataset compared with baseline powerfully demonstrate the effectiveness and superiority of the SGFTHR network.

The binary hypothesis testing (BHT) is one of the most important models in hyperspectral target detection (HTD). However, this model is generally based on a linear mixture model (LMM) and might be inaccurate to reflect target and background characterizations in some scenes. This article presents a bilinear sparse target detector (BSTD) by applying the bilinear sparse mixture model (BSMM) to a popular BHT-based detection algorithm termed adaptive matched subspace detector (AMSD), which takes bilinear target–background interaction and sparse abundance into account. Moreover, as AMSD relies heavily on background subspace, we design a robust background subspace construction method. Specifically, we first classify each pixel into noise, border, or other particular instances according to its density, which is measured by jointly spatial–spectral distance. With the coarse classification map, a class-guided automatic background generation (CABG) process is introduced to reliably generate pure background samples. Detection statistics and component analysis on five real-world hyperspectral images verify the effectiveness of our BSTD method.

In the above article[1],Fig. 19was incorrectly placed. The correct image and caption are provided here:

Convolutional neural networks (CNNs) have achieved prominent progress in recent years and demonstrated remarkable properties in spectral–spatial hyperspectral image (HSI) classification. However, conventional spatial-context-based CNNs commonly adopt the single patchwise scheme to represent the to-be-classified samples, which often fails to completely investigate the wealthy spectral–spatial information in complicated situations. For instance, it has great probability to cause misclassifications on the irregular or inhomogeneous areas, especially for the borders across different classes. To counteract this deficiency, we propose a unified multi-direction network (MDN) for HSI Classification (HSIC), which can exhaustively explore the abundant spectral and detailed spatial-context information through multi-direction samples. Additionally, considering the image-spectrum merged structure of the HSI, 3-D Squeeze-and-Excitation residual (3DSERes) blocks are devised in each stream of the framework to consecutively learn the spectral and spatial from low-level to high-level features. Specifically, 3DSERes can not only facilitate fluent gradient in backpropagation through skip connections, but also emphasize the significant spectral–spatial features and constrain the futile ones. This characteristic is beneficial to enhance the model’s generalization capability even with limited training samples. Furthermore, for properly aggregating the multi-direction deep features, we exploit the simple, yet effective attentional spectral prior (ASP) creatively through leveraging the original spectral correlations. Extensive experimental results on three benchmark data sets indicate that the proposed MDN-ASP can achieve promising classification performance compared to the state-of-the-art methods.

Despite recent advances achieved by deep learning techniques in the fusion of low-spatial-resolution hyperspectral image (LR-HSI) and high-spatial-resolution multispectral image (HR-MSI), it remains a challenge to reconstruct the high-spatial-resolution HSI (HR-HSI) with more accurate spatial details and less spectral distortions, since the low-level structure information such as sharp edges tends to be weakened or lost as the network depth grows. To tackle this issue, we creatively propose an edge-conditioned feature transform network (EC-FTN) in this article, which is mainly composed of three parts, namely, feature extraction network (FEN), feature fusion and transformation network (FFTN), and image reconstruction network (IRN). First, two computationally efficient FENs with 3-D convolutions and reshaping layers are employed to extract the joint spectral-spatial features of input images. Then, the FFTN conditioned on the edge map prior can fuse and transform the features adaptively, in which a fusion node and several cascaded feature modulation modules (FMMs) equipped with feature-wise modulation layers are constructed. Specifically, the edge map is generated via transfer learning, i.e., by applying the Sobel operator to feature maps of the red-green-blue (RGB) version of HR-MSI resulting from the pretrained VGG16 model without extra training. Finally, the desired HR-HSI is recovered from the transformed features through IRN. Furthermore, we elaborately design a weighted combinatorial loss function consisting of mean absolute error, image gradient difference, and spectral angle terms to guide the training. Experiments on both ground-based and remotely sensed datasets demonstrate that our EC-FTN outperforms state-of-the-art methods in visual and quantitive evaluations, as well as in fine details reconstruction.

Hyperspectral imagery with very high spectral resolution provides a new insight for subtle nuances identification of similar substances. However, hyperspectral target detection faces significant challenges of intraclass dissimilarity and interclass similarity due to the unavoidable interference caused by atmosphere, illumination, and sensor noise. In order to effectively alleviate these spectral inconsistencies, this paper proposes a novel target detection method without strict assumptions on data distribution based on an unconstrained linear mixture model and deep learning. Our proposed detector firstly reduces interference via a specifically designed deep-learning-based hierarchical denoising autoencoder, and then carries out accurate detection with a two-step subspace projection, aiming at background suppression and target enhancement. Additionally, to generate representative background and reliable target samples required in the detection procedure, an efficient spatial-spectral unified endmember extraction method has been developed. Performance comparison with several state-of-the-art detection methods and further analysis on four real-world hyperspectral images demonstrate the effectiveness and efficiency of our proposed target detector.

Recently, embedding and metric-based few-shot learning (FSL) has been introduced into hyperspectral image classification (HSIC) and achieved impressive progress. To further enhance the performance with few labeled samples, we in this paper propose a novel FSL framework for HSIC with a class-covariance metric (CMFSL). Overall, the CMFSL learns global class representations for each training episode by interactively using training samples from the base and novel classes, and a synthesis strategy is employed on the novel classes to avoid overfitting. During the meta-training and meta-testing, the class labels are determined directly using the Mahalanobis distance measurement rather than an extra classifier. Benefiting from the task-adapted class-covariance estimations, the CMFSL can construct more flexible decision boundaries than the commonly used Euclidean metric. Additionally, a lightweight cross-scale convolutional network (LXConvNet) consisting of 3D and 2D convolutions is designed to thoroughly exploit the spectral-spatial information in the high-frequency and low-frequency scales with low computational complexity. Furthermore, we devise a spectral-prior-based refinement module (SPRM) in the initial stage of feature extraction, which cannot only force the network to emphasize the most informative bands while suppressing the useless ones, but also alleviate the effects of the domain shift between the base and novel categories to learn a collaborative embedding mapping. Extensive experiment results on four benchmark data sets demonstrate that the proposed CMFSL can outperform the state-of-the-art methods with few-shot annotated samples.

The fusion framework of star sensor and gyro based on adaptive Kalman filter is widely used in satellite pose estimation. However, the discretization and linearization inevitably introduce system errors, which degrades of the filtering accuracy. To address this problem, we propose a high-precision satellite attitude determination algorithm based on extreme learning machine network correction. We design a dedicated network for error compensation and trained the parameters effectively. In attitude calculation procedure, the forward fusion filtering of star sensor and gyro data is performed firstly by using the adaptive Kalman filter. Then the filtering estimation results are compensated by the extreme learning machine network proposed in this paper. After that, backward smoothing is performed to solve the high-precision attitude. Simulation results show that armed with the compensation procedure of the proposed extreme learning machine network, the accuracy of estimated pose is significantly improved.

Target detection technique in hyperspectral imagery has been widely applied in various applications. However, its performance is severely limited by the useless interference contained in hyperspectral images (HSIs), mainly caused by the atmosphere, illumination, issues within the sensor itself, and some other factors. In this paper, we propose a hyperspectral target detector based on linear mixture model (LMM), which consists of three components. First, a hierarchical denoising autoencoder (HDAE) is specifically designed for redundant interference removal; then we apply an adaptive cluster approach to extract several representative background samples from the clean HSI; lastly, a target detector with subspace projection is developed for background suppression and target enhancement based on the clean HSI, representative background and prior-known target signatures. Experimental results on two real-world HSIs show the superiority of our proposed method, namely, the HDASP detector, comparing with other state-of-the-art target detection methods.

Recently, deep convolutional neural networks (CNNs) have been widely exploited for spectral reconstruction (SR) and achieved significant promotion. Nevertheless, most of the previous studies paid much attention to the design of the depth and width of the network, and neglected to explore the correlation between the intermediate feature maps, which hindered the representational ability of CNNs. To mitigate this problem, we propose a deep residual channel affinity propagation network (RCAPN) to learn the affinity matrix for more powerful feature expression. Specifically, the backbone consists of several dual residual blocks (DRB) with long and short skip connections to bypass plentiful low-frequency information. Furthermore, a novel channel affinity propagation module (CAPM) embedded in the DRB is investigated to learn the affinity among channels and adaptively integrate interdependencies to strengthen feature representations. Finally, a structural similarity (SSIM) constraint is employed to capture the structural information and recover more accurate edge positions. Experimental results demonstrate the superior performance of our proposed algorithm.

Graph convolutional network (GCN) is one of the most favorable semi-supervised approaches, which demonstrates encouraging performance for hyperspectral image classification (HSIC), especially under the condition of small sample sizes. In this paper, we propose a novel semi-supervised graph prototypical network (SSGPN) for high-precise HSIC. Different from prevenient GCN, we devise a prototypical layer comprising a distance-based cross-entropy (DCE) loss function and a novel temporal entropy-based regularizer (TER) in the frameworks of SSGPN. This effective layer can facilitate to generate more discriminative embedding features along with the representative prototypes to each class, so as to achieve accurate identification of various land-cover categories. Additionally, to promote computational efficiency, we present a graph normalization (G-Norm) to accelerate the convergence speed and boost the training procedure. Experimental results demonstrate that our proposed SSGPN can obtain promising performance compared with the state-of-the-art methods.

In recent years, object detection in remote sensing imagery has attracted more and more attention. Accurate object detection in remote sensing imagery, especially for small objects, is still challenging. Most existing methods utilize the global information in the deep fully convolution layer and neglect the local information in the input image. However, the local information contains sufficient spatial information, which is beneficial for precise localization. Additionally, there still exists variable factors, such as the arbitrary aspect ratio and rotation, which interfere the object detection performance. To solve these problems mentioned above, we propose a novel multi-scale structure-conditioned feature transform network, which adopts FCOS as the baseline and ATSS as the method for training sample selection. On one hand, structural information is extracted to represent the spatial semantic information. On the other hand, multi-scale information is enhanced through a novel hierarchical residual-in-residual module. Experiments on the HRRSD data set have demonstrated the superiority of our method.

Pansharpening of hyperspectral image (HSI), which makes use of the detail information contained in the high-resolution panchromatic (HR-PAN) image to sharpen the low-resolution HSI (LR-HSI), is an essential technology to enhance the spatial resolution of HSI. In this paper, we propose a detail guided feature modulation residual network (DGFM-Net) to address the HS pansharpening problem, which is able to effectively integrate details extracted from the PAN image into the pansharpened result. Specifically, we elaborately design a novel feature modulation (FM) module with the guidance of PAN detail information to modulate HSI features flexibly and incorporate PAN details adaptively. The modulated features are then fed to the residual reconstruction (RR) block to recover the difference between the upsampled HSI and the HR-HSI by efficient residual learning. Finally, the upsampled HSI is combined with the estimated residual HSI to produce the desired HR-HSI. Experiments on the Pavia Center data set confirm that the proposed DGFM-Net outperforms several state-of-the-art HS pansharpening methods.

In this article, we propose a weakly supervised low-rank representation (WSLRR) method for hyperspectral anomaly detection (HAD), which formulates deep learning-based HAD into a low-lank optimization problem not only characterizing the complex and diverse background in real HSIs but also obtaining relatively strong supervision information. Different from the existing unsupervised and supervised methods, we first model the background in a weakly supervised manner, which achieves better performance without prior information and is not restrained by richly correct annotation. Considering reconstruction biases introduced by the weakly supervised estimation, LRR is an effective method for further exploring the intricate background structures. Instead of directly applying the conventional LRR approaches, a dictionary-based LRR, including both observed training data and hidden learned data drawn by the background estimation model, is proposed. Finally, the derived low-rank part and sparse part and the result of the initial detection work together to achieve anomaly detection. Comparative analyses validate that the proposed WSLRR method presents superior detection performance compared with the state-of-the-art methods.

RGB image spectral super-resolution (SSR) is a challenging task due to its serious ill-posedness, which aims at recovering a hyperspectral image (HSI) from a corresponding RGB image. In this article, we propose a novel hybrid 2-D-3-D deep residual attentional network (HDRAN) with structure tensor constraints, which can take fully advantage of the spatial-spectral context information in the reconstruction progress. Previous works improve the SSR performance only through stacking more layers to catch local spatial correlation neglecting the differences and interdependences among features, especially band features; different from them, our novel method focuses on the context information utilization. First, the proposed HDRAN consists of a 2D-RAN following by a 3D-RAN, where the 2D-RAN mainly focuses on extracting abundant spatial features, whereas the 3D-RAN mainly simulates the interband correlations. Then, we introduce 2-D channel attention and 3-D band attention mechanisms into the 2D-RAN and 3D-RAN, respectively, to adaptively recalibrate channelwise and bandwise feature responses for enhancing context features. Besides, since structure tensor represents structure and spatial information, we apply structure tensor constraint to further reconstruct more accurate high-frequency details during the training process. Experimental results demonstrate that our proposed method achieves the state-of-the-art performance in terms of mean relative absolute error (MRAE) and root mean square error (RMSE) on both the “clean” and “real world” tracks in the NTIRE 2018 Spectral Reconstruction Challenge. As for competitive ranking metric MRAE, our method separately achieves a 16.06% and 2.90% relative reduction on two tracks over the first place. Furthermore, we investigate HDRAN on the other two HSI benchmarks noted as the CAVE and Harvard data sets, also demonstrating better results than state-of-the-art methods.

Deep learning-based hyperspectral target detection (HTD) is potentially hindered by the limited training samples and sensor-dependent transferability. To address this issue, we propose a novel semisupervised domain adaptive few-shot learning (SDAFL) model to adaptively transfer similarity/dissimilarity measurement from source domain with sufficient labeled samples to target domain in an adversarial manner, where source data and target data can be collected by different sensors, i.e., sensor-independent. In order to alleviate negative transfer, residual channel attention (RCA) and weighted domain adaptation (WDA) are used to automatically select representative features and assign easy-transferred samples with higher priority. In addition, we adopt modulated deformable convolution (MDConv) to make the receptive field fit image spatial structure and also introduce a discriminatively boosted loss (DBL) function based on the prior known target signature to further enhance feature distinction, where intraclass similarity is improved, while interclass similarity is suppressed. After extracting discriminative features through the SDAFL model, guided filter and t-distribution kernel are jointly used for spatial–spectral target detection (S2TD). It should be noted that only the spectral signature of the desired object is needed in the target domain. Experimental results and analysis on three real hyperspectral images (HSIs) verify the efficiency and superiority of our proposed sensor-independent hyperspectral target detection (SIHTD) method compared with other algorithms.

Target detection plays a core issue in hyperspectral remote sensing, but faces serious challenges of how to deal with the spatial and spectral redundancies and spectral variations. In this article, a novel network block is developed, called RFT-MSA block (abbreviated as RM), which includes the region-of-interest (RoI) feature transformation (RFT) and the multiscale-spectral-attention (MSA) module as to reduce the spatial and spectral redundancies simultaneously and provide strong discrimination. Furthermore, a deep spatial-spectral network (DSSN) is presented by stacking several RM and deconvolutional (DC) blocks for hyperspectral target detection in an unsupervised manner, and a feature loss term is investigated to simultaneously restrict the target to be sparse and minimize the energy of the background. The proposed algorithm mainly consists of three steps. First, an RoI map is detected using a classical detector (no statistic assumption is needed) with an edge-preserving filter. Then, the hyperspectral image (HSI) and the corresponding RoI map are considered as inputs to the DSSN for extracting the spatial and spectral feature of interest (SSFI). Finally, we apply the nearest neighbors (NNs) to the SSFI for detection-map refinement. The experimental results on one synthetic and three real HSIs demonstrate that the proposed algorithm outperforms other benchmark approaches in detection performance and robustness. In addition, further analysis also demonstrates the effectiveness of the proposed RM block.

Recently, multiscale spatial features have been widely utilized to improve the hyperspectral image (HSI) classification performance. However, fixed-size neighborhood involving the contextual information probably leads to misclassifications, especially for the boundary pixels. Additionally, it has been demonstrated that deep neural network (DNN) is practical to extract representative features for the classification tasks. Nevertheless, under the condition of high dimensionality versus small sample sizes, DNN tends to be over-fitting and it is generally time-consuming due to the deep-level feature learning process. To alleviate the aforementioned issues, we propose a multiscale context-aware ensemble deep kernel extreme learning machine (MSC-EDKELM) for efficient HSI classification. First, the scene of the HSI data set is over-segmented in multiscale via using the adaptive superpixel segmentation technique. Second, superpixel pattern (SP) and attentional neighboring superpixel pattern (ANSP) are generated by leveraging the superpixel maps, which can automatically comprise local and global contextual information, respectively. Afterward, an ensemble deep kernel extreme learning machine (EDKELM) is presented to investigate the deep-level characteristics in the SP and ANSP. Finally, the category of each pixel is accurately determined by the decision fusion and weighted output layer fusion strategy. Experimental results on four real-world HSI data sets demonstrate that the proposed frameworks outperform some classic and state-of-the-art methods with high computational efficiency, which can be employed to serve real-time applications.

Hyperspectral (HS) pansharpening, as a special case of the superresolution (SR) problem, is to obtain a high-resolution (HR) image from the fusion of an HR panchromatic (PAN) image and a low-resolution (LR) HS image. Though HS pansharpening based on deep learning has gained rapid development in recent years, it is still a challenging task because of the following requirements: 1) a unique model with the goal of fusing two images with different dimensions should enhance spatial resolution while preserving spectral information; 2) all the parameters should be adaptively trained without manual adjustment; and 3) a model with good generalization should overcome the sensitivity to different sensor data in reasonable computational complexity. To meet such requirements, we propose a unique HS pansharpening framework based on a 3-D generative adversarial network (HPGAN) in this article. The HPGAN induces the 3-D spectral-spatial generator network to reconstruct the HR HS image from the newly constructed 3-D PAN cube and the LR HS image. It searches for an optimal HR HS image by successive adversarial learning to fool the introduced PAN discriminator network. The loss function is specifically designed to comprehensively consider global constraint, spectral constraint, and spatial constraint. Besides, the proposed 3-D training in the high-frequency domain reduces the sensitivity to different sensor data and extends the generalization of HPGAN. Experimental results on data sets captured by different sensors illustrate that the proposed method can successfully enhance spatial resolution and preserve spectral information.

Compared with common RGB images, remote sensing images (RSIs) have larger size and lower spatial resolution. RSIs are usually cropped into sub-images for training convolutional neural networks (CNNs), which loses amounts of context information, thus limiting the extraction of feature interdependencies and reducing the accuracy of semantic segmentation. In this paper, a novel dilated residual network based on dual expectation maximization attention (DE-MANet) is proposed for semantic segmentation of RSIs. In specific, we append a dual expectation maximization attention (DEMA) module on top of the dilated CNN. The spatial expectation maximization attention (SEMA) can model spatial feature interdependencies to acquire rich long-range contextual information. The channel expectation maximization attention (CEMA) enhances discriminant ability of channel-wise feature representations through extracting the channel dependencies. We evaluate the model on the dataset released in the Tianzhi Cup Artificial Intelligence Challenge and achieve 85.60% pixel accuracy and 69.00% mean intersection over union (mIoU).

Hyperspectral target detection has been widely used in practice, but its performance is seriously affected by spatial redundancy and spectral variation. In this paper, we design a novel network block including the region of interest (RoI) feature transformation (RFT) and convolution layer, which is called RS block, that can automatically attach different importance to pixels and provide guidance to discriminative feature extraction. Furthermore, a deep neural network (termed as RFTD) is proposed by stacking several RS blocks for hyperspectral target detection. In this way, the RS block enforces RFTD to concentrate on RoI feature extraction, and further increase distinction between target and hard-detected background (false alarm) pixels. Additionally, a constraint loss is introduced to exploit the sparsity and low rank property of hyperspectral images (HSI). Finally, we apply nearest neighbors (NN) for target detection in the feature space. Experimental results on two HSIs demonstrate that the proposed RFTD algorithm outperforms other detection methods.

With the development of deep convolutional neural networks (CNNs), spectral super-resolution (SSR) has obtained a significant improvement, which aims to recover the hyperspectral image (HSI) from a single RGB. However, the existing mapping algorithms lack of utilization of the camera spectral sensitivity (CSS) and only focus on wider or deeper architecture design, neglecting to explore the feature correlations of intermediate layers, thus preventing the representational ability of CNNs. In our paper, a novel hybrid 2D-3D structure tensor attention networks (HSTAN) with CSS prior is proposed for SSR. In specific, a structure tensor attention (STA) embedded in the residual block is invented to extract the salient high-frequency spatial details for adequate spatial feature expression. Furthermore, the CSS is firstly exploited as a prior to avoid its influence of SSR quality, based on which the reconstructed RGB can be calculated naturally through the super-resolved HSI, then the final loss incorporates the discrepancies of RGB and the HSI as a finer constraint. Experimental results demonstrate the superiority of our proposed algorithm.

In this paper, we propose a deep residual spatial attention network (DRSAN) for hyperspectral (HS) pansharpening. Different from the existing methods, our newly proposed method not only considers the spatial information of both the panchromatic (PAN) and the HS image simultaneously, but also adaptively learns more informative features of spatial locations for details enhancement, which mainly includes four steps. Firstly, the spatial details of the enhanced PAN image are obtained through the structure tensor. Then we extract the spatial information of the upsampled HSI by using the guided filter. The integrated spatial information of both PAN and HS images is subsequently fed into the DRSAN to map the residual HSI between the upsampled HSI and the reference HSI, where several residual spatial attention blocks (RSABs) are cascaded to exploit more useful details information. Finally, the fused HSI is generated by the summation of the upsampled HSI and the reconstructed residual HSI. Extensive visual and quantitative assessments validate the superiority of our proposed DRSAN over the state-of-the-art HS pansharpening methods.

Recently, deep learning (DL) has gained impressive achievements in the field of remote sensing image fusion. However, most of the previous DL-based fusion methods are originally designed for multispectral pansharpening, which cannot be readily employed to hyperspectral pansharpening due to the much wider spectral range and lower spatial resolution of a hyperspectral image (HSI). In this letter, a novel framework based on deep residual learning is proposed for hyperspectral pansharpening. The proposed framework consists mainly of two parts. First, the initialized HSI with the enhanced spatial resolution is generated through contrast limited adaptive histogram equalization (CLAHE) and guided filter. Then, a deep residual convolutional neural network (DRCNN) is introduced to map the residuals between the initialized HSI and the reference HSI for further boosting the fusion accuracy. Experimental results demonstrate that the proposed framework can achieve superior performance compared with the existing state-of-the-art pansharpening methods, especially in terms of edge details enhancement.

Hyperspectral image (HSI) super-resolution (SR) is a challenging task due to the problems of texture blur and spectral distortion when the upscaling factor is large. To meet these two challenges, band attention through the adversarial learning method is proposed in this article. First, we put the SR process in a generative adversarial network (GAN) framework, so that the resulted high-resolution HSI can keep more texture details. Second, different from the other band-by-band SR method, the input of our method is of full bands. In order to explore the correlation of spectral bands and avoid the spectral distortion, a band attention mechanism is proposed in our generative network. A series of spatial-spectral constraints or loss functions is imposed to guide the training of our generative network so as to further alleviate spectral distortion and texture blur. The experiments on the Pavia and Cave data sets demonstrate that the proposed GAN-based SR method can yield very high-quality results, even under large upscaling factor (e.g., $8\times $ ). More importantly, it can outperform the other state-of-the-art methods by a margin which demonstrates its superiority and effectiveness.

Convolutional neural networks (CNNs) have recently achieved impressive improvements on hyperspectral (HS) pansharpening. However, most of the CNN-based HS pansharpening approaches would have to first upsample the low-resolution hyperspectral image (LR-HSI) using bicubic interpolation or data-driven training strategy, which inevitably lose some details or greatly rely on the learning process. In addition, most previous methods regard the pansharpening as a black-box problem and treat diverse features equally, thus hindering the discriminative ability of CNNs. To conquer these issues, a novel HS pansharpening method using deep hyperspectral prior (DHP) and dual-attention residual network (DARN) is proposed in this article. Specifically, we first upsample the LR-HSI to the scale of the panchromatic (PAN) image through the DHP algorithm, which can better preserve spatial and spectral information without learning from large data sets. The upsampled result is then concatenated with the PAN image to form the input of the DARN, where several channel-spatial attention residual blocks (CSA ResBlocks) are stacked to map the residual HSI between the reference HSI and the upsampled HSI. In each CSA ResBlock, two complementary attention modules, i.e., channel attention and spatial attention modules, are designed to adaptively learn more informative features of spectral channels and spatial locations simultaneously, which can effectively boost the fusion accuracy. Finally, the fused HSI is obtained by the summation of the upsampled HSI and the reconstructed residual HSI. The experimental results of both simulated and real HS data sets demonstrate that the performance of our DHP-DARN method is superior over the state-of-the-art HS pansharpening approaches.

Hyperspectral image (HSI) spatial super-resolution(SR) is a challenging task. Compared with a RGB images, the mapping between the low-high HSI pairs is more difficult since much more spectral bands are involved. In this paper, a novel dual 1D-2D spatial-spectral convolutional neural network (CNN) architecture is proposed for spatial SR of HSIs. Specifically, by differential treatment over redundancy in spectral and spatial domains of an HSI, the spectral and spatial context are first separately explored by 1D and 2D convolution. These two kinds of feature information are then fused using a novel hierarchical side connection, which impose the spectral information to the spatial path gradually. Experimental results over benchmark Pavia data set demonstrate that the proposed architecture clearly outperform state-of-the-art 3D CNN based works in terms of both visual quality and quantitative assessment.

Recently, deep learning technology has gained impressive effectiveness in the field of hyperspectral pansharpening. However, the existing methods with relatively shallow architectures ignores the deep features of hyperspectral image (HSI) and panchromatic (PAN) image, which leads to a limitation of the fusion performance. To address this issue, a novel hyperspectral pansharpening framework based on guided filter and deep residual learning is proposed in this paper. The proposed framework mainly consists of two parts: generating the initial HSI through enhancing spatial information while preserving the original spectral information, and mapping the residuals between the initialized HSI and the reference HSI for further improvement of the fusion accuracy. Experimental results demonstrate that the proposed framework can achieve superior fusion accuracy compared with other state-of-the-art hyperspectral pansharpening methods while providing better edge information.

Spatial context has been demonstrated to be effective to constrain sparse unmixing (SU) of hyperspectral images. However, the existing algorithms employed simple spatial information without keeping spectral fidelity. By considering the fact that adjacent pixels own not only the endmembers with same variations but also approximated fractional abundances, in this paper, local spectral similarity preserving (LSSP) constraint is proposed to preserve spectral similarity in a local area during robust sparse unmixing (RSU). Specially, four LSSP constraints are constructed using different-norm-constrained pixel-level difference over abundance-level difference in a local area. Moreover, a convex optimization algorithm is proposed to solve the proposed LSSP-constrained RSU (LSSP-RSU). Experimental results on both synthetic and real hyperspectral data demonstrate that the developed algorithms yield better values of the signal-toreconstruction error (SRE). Especially, when using l2norm of pixel-level difference to weight the l1norm of abundance-level difference, the proposed LSSP-RSU algorithm can achieve superior unmixing performance.

With success of convolutional neural networks (CNNs) in computer vision, the CNN has attracted great attention in hyperspectral classification. Many deep learning-based algorithms have been focused on deep feature extraction for classification improvement. In this letter, a novel deep learning framework for hyperspectral classification based on a fully CNN is proposed. Through convolution, deconvolution, and pooling layers, the deep features of hyperspectral data are enhanced. After feature enhancement, the optimized extreme learning machine (ELM) is utilized for classification. The proposed framework outperforms the existing CNN and other traditional classification algorithms by including deconvolution layers and an optimized ELM. Experimental results demonstrate that it can achieve outstanding hyperspectral classification performance.

Conventional classification algorithms have shown great success for balanced classes. In remote sensing applications, it is often the case that classes are imbalanced. This paper proposes a novel solution to solve the problem of imbalanced training samples in hyperspectral image classification. It consists of two parts: one is for large-size sample sets and the other is for small-size sets. Specifically, an algorithm based on the orthogonal complement subspace projection (OCSP) is proposed to select samples from large-size classes, and an algorithm also based on OCSP is proposed to create artificial samples for small-size ones. The impact on representation-based classifiers, i.e., sparse and collaborative representation classifiers and traditional classifiers (e.g., support vector machine), is investigated. Experimental results demonstrate that the proposed solution can outperform other existing solutions in the literature.

Haze removal for a single image is known to be a challenging ill-posed problem in computer vision. The performance of existing prior-based image dehazing methods is limited by the effectiveness of hand-designed features. The emerging con-volutional neural network (CNN) based approaches can remove haze with the automatically learned intrinsic mapping between the input hazy images and their corresponding transmission maps, but the recovered haze-free images sometimes are still unsatisfactory. In order to improve the dehazed images, we aim to develop an effective deep fully convolutional regression network for more accurate transmission estimation. Our network is an end-to-end regression system which take input of arbitrary size hazy image and predict correspondingly-sized transmission map. To train and evaluate deep network for image dehazing efficiently, we develop new outdoor synthetic training set respectively. In addition, we fully compare the existing CNN-based haze removal approaches with our algorithm on real-world images and our synthesized benchmark dataset. The experimental results demonstrate that our trained regression model achieves superior dehazing performance than the current state-of-the-art methods.

Hyperspectral (HS) pansharpening aims at fusing a panchromatic (PAN) image with a hyperspectral image, generating an image with the high spatial resolution of the former and the high spectral resolution of the latter. Recently, in order to enhance this process, researches have combined hyperspectral unmixing with the HS fusion model, improving the fusion results . In this combined model , endmember subspace extraction is a crucial step. Traditionally, VCA and SVD are used for subspace extraction, but VCA extracts pixels in the HS image that contain impure material signatures. In this work, we use the sparse unmixing technique to extract the endmember subspace which contains pure material signatures, which may better represent the HS image. We combine sparse unmixing-based subspace extraction with the bayesian fusion model. Results indicate that the fusion algorithm using our subspace extraction method had better global performance.

This paper proposes a novel solution to solve the problem of imbalanced training samples in hyperspectral image classification. It consists of two parts: one is for large-size sample sets and the other is for small-size sets. We exploit an orthogonal projection based algorithm to select samples from large-size ones; meanwhile, we propose an algorithm based on the orthogonal complementary subspace projection to create artificial samples for small-size ones. The impact on representation based classifiers, i.e., sparse representation based classifier and collaborative representation based classifier, are investigated. Experimental results demonstrate that it can outperform other traditional solutions.