Despite the rapid progress of multimodal large language models (MLLMs), their capacity for low-level visual perception in underwater environments remains underexplored. To address this gap, we present UQ-Bench, the first systematically designed benchmark for evaluating the ability of MLLMs to perceive and assess underwater image quality at the low-level visual attribute level. UQ-Bench comprises three components: (1) UW-Perception, a dataset of 3,000 underwater images paired with targeted questions on key degradations such as color cast, blur, contrast, and exposure, covering both global and local perceptual dimensions; (2) UW-Describe, a dataset of 500 images with expert-annotated gold-standard descriptions for assessing the accuracy of model-generated text; and (3) UW-Eval, an evaluation protocol employing human mean opinion scores (MOS) for quantitative quality assessment. To ensure rigorous and reproducible benchmarking, we propose a GPT-assisted evaluation framework that aligns model outputs with expert references and enables fine-grained analysis of distortion perception. Experimental results demonstrate that while MLLMs exhibit preliminary competence in underwater low-level visual tasks, they still fall short in capturing subtle degradations and achieving human-level consistency, highlighting the need for further advances in foundation models for marine vision.

Purple fringing, a persistent artifact caused by Longitudinal Chromatic Aberration (LCA) in camera lenses, has long degraded the clarity and realism of digital imaging. Traditional solutions rely on complex and expensive apochromatic (APO) lens hardware and the extraction of handcrafted features, ignoring the data-driven approach. To fill this gap, we introduce DCA-LUT, the first deep learning framework for purple fringing removal. Inspired by the physical root of the problem-the spatial misalignment of RGB color channels due to lens dispersion, we introduce a novel Chromatic-Aware Coordinate Transformation (CA-CT) module, learning an image-adaptive color space to decouple and isolate fringing into a dedicated dimension. This targeted separation allows the network to learn a precise "purple fringe channel," which then guides the accurate restoration of the luminance channel. The final color correction is performed by a learned 5D Look-Up Table (5D LUT), enabling efficient and powerful non-linear color mapping. To enable robust training and fair evaluation, we constructed a large-scale synthetic purple fringing dataset (PF-Synth). Extensive experiments in synthetic and real-world datasets demonstrate that our method achieves state-of-the-art performance in purple fringing removal.

The scarcity of paired data severely limits the performance and generalization of learning-based underwater image enhancement (UIE) methods. This challenge is particularly prominent in scenes with complex degradations. Semi-supervised learning has emerged as a promising solution by enabling the utilization of large-scale unlabeled data. However, its effectiveness is limited by the use of static, model-agnostic metrics for pseudo-label reliability assessment. To address this, we propose SEA-PACE, a novel semi-supervised framework that integrates model-aware uncertainty modeling and self-paced consistency learning to fully exploit unlabeled data for UIE. Specifically, we design a Model-Aware Reliability Estimator (MARE) that quantifies the uncertainty of the teacher model's predictions through Gaussian Process Regression in latent feature space. The resulting uncertainty is then transformed into reliability weights via a rank-based mapping. Additionally, we apply the Self-Paced Consistency Learning (SPCL) strategy that employs a loss-aware schedule to dynamically prioritize high-confidence pseudo-labels, gradually incorporating more challenging samples during training. Extensive experiments on several public UIE benchmarks demonstrate that SEA-PACE consistently surpasses state-of-the-art methods in both visual quality and generalization capability.

Accurate 3D hand pose estimation faces inherent challenges, including self-occlusions, joint similarities, and high degrees of freedom. Although most existing CNN-based or Transformer-based methods leverage global contexts, they often fail to capture fine-grained local details and robustly handle occluded joints. To address these limitations, we propose a novel Transformer framework for depth-based 3D hand pose estimation, which incorporates two key designs: the Bidirectional Multiscale Fusion and Learnable Anchor Guidance. Firstly, we propose bidirectional multiscale fusion that sequentially propa-gates features from different encoder levels in both top-down and bottom-up directions, followed by a final aggregation of all scale features. Such an intricate feature interaction eventually enables effective joint modeling of fine-grained local details (e.g., fingertip positions) and high-level semantic context information (e.g., palm orientation). Secondly, we introduce the learnable anchor query as prior guidance to dynamically guide the decoder to localize ambiguous joints better. The learnable anchors are derived from joint-specific attention maps under 3D ground-truth supervision and then are concatenated with static grid anchors to form hybrid anchors, which effectively enable more precise 3D hand pose estimation, especially for occlusions. To further demonstrate the practicality of our framework for IoT-oriented deployment, we conduct edge-device experiments to validate its deployment feasibility. Experiments on benchmark datasets (including NYU, ICVL, MSRA, and DexYCB) demonstrate the superiority of our method over previous state-of-the-art approaches.

In real underwater environments, downstream image recognition tasks such as semantic segmentation and object detection often face challenges posed by problems like blurring and color inconsistencies. Underwater image enhancement (UIE) has emerged as a promising preprocessing approach, aiming to improve the recognizability of targets in underwater images. However, most existing UIE methods mainly focus on enhancing images for human visual perception, frequently failing to reconstruct high-frequency details that are critical for task-specific recognition. To address this issue, we propose a Downstream Task-Inspired Underwater Image Enhancement (DTI-UIE) framework, which leverages human visual perception model to enhance images effectively for underwater vision tasks. Specifically, we design an efficient two-branch network with task-aware attention module for feature mixing. The network benefits from a multi-stage training framework and a task-driven perceptual loss. Additionally, inspired by human perception, we automatically construct a Task-Inspired UIE Dataset (TI-UIED) using various task-specific networks. Experimental results demonstrate that DTI-UIE significantly improves task performance by generating preprocessed images that are beneficial for downstream tasks such as semantic segmentation, object detection, and instance segmentation. The code will be made publicly available at https://github.com/oucailab/DTIUIE.

Change detection from synthetic aperture radar (SAR) imagery is critical in remote sensing research. Existing methods have made significant progress in the application of convolutional neural networks (CNNs) and attention mechanisms. However, traditional CNNs suffer the limitations in feature representation due to their depth and width constraints, and struggle to effectively capture complex interactions between image features. To address these issues, we propose a novel dynamic bilinear fusion network (DBFNet) for change detection in SAR imagery. First, to compensate for the lack of traditional convolutional representation capability, we design a dynamic shift convolution module that adaptively aggregates multiple convolution kernels and shifts pixels, enabling richer and more detailed features to be extracted. Second, a bilinear fusion module (BFM) is designed to generate the bilinear joint representation between parallel features by computing a matrix outer product of feature maps. The parallel features include both intraimage and interimage features, thereby effectively modeling the complex interactions and capturing the dependence relationship between spatiotemporal features. The experimental results on three real SAR datasets demonstrate the superior performance of DBFNet compared to existing state-of-the-art methods. The codes are available athttps://github.com/yizhilanmaodhh/DBFNet.

Recently, transformer-based hyperspectral image (HSI) change detection methods have shown remarkable performance. Nevertheless, existing attention mechanisms in transformers have limitations in local feature representation. To address this issue, we propose global and local attention-based transformer (GLAFormer), which incorporates a global and local attention module (GLAM) to combine high-frequency and low-frequency signals. Furthermore, we introduce a cross-gating mechanism, called cross-gated feedforward network (CGFN), to emphasize salient features and suppress noise interference. Specifically, the GLAM splits attention heads into global and local attention components to capture comprehensive spatial–spectral features. The global attention component uses global attention on downsampled feature maps to capture low-frequency information, while the local attention component focuses on high-frequency details using nonoverlapping window-based local attention. The CGFN enhances the feature representation via convolutions and cross-gating mechanism in parallel paths. The proposed GLAFormer is evaluated on three HSI datasets. The results demonstrate its superiority over state-of-the-art HSI change detection methods. The source code of GLAFormer is available athttps://github.com/summitgao/GLAFormer.

Multi-source data classification is a critical yet challenging task for remote sensing image interpretation. Existing methods lack adaptability to diverse land cover types when modeling frequency domain features. To this end, we propose a Dynamic Frequency Feature Fusion Network (DFFNet) for hyperspectral image (HSI) and Synthetic Aperture Radar (SAR) / Light Detection and Ranging (LiDAR) data joint classification. Specifically, we design a dynamic filter block to dynamically learn the filter kernels in the frequency domain by aggregating the input features. The frequency contextual knowledge is injected into frequency filter kernels. Additionally, we propose spectral-spatial adaptive fusion block for cross-modal feature fusion. It enhances the spectral and spatial attention weight interactions via channel shuffle operation, thereby providing comprehensive cross-modal feature fusion. Experiments on two benchmark datasets show that our DFFNet outperforms state-of-the-art methods in multi-source data classification. The codes will be made publicly available at https://github.com/oucailab/DFFNet.

Image enhancement plays a crucial role in computer vision by improving visual quality while minimizing distortion. Traditional methods enhance images through pixel value transformations, yet they often introduce new distortions. Recent advancements in deep learning-based techniques promise better results but challenge the preservation of image fidelity. Therefore, it is essential to evaluate the visual quality of enhanced images. However, existing quality assessment methods frequently encounter difficulties due to the unique distortions introduced by these enhancements, thereby restricting their effectiveness. To address these challenges, this paper proposes a novel blind image quality assessment (BIQA) method for enhanced natural images, termed multi-scale local feature fusion and global feature representation-based quality assessment (MLGQA). This model integrates three key components: a multi-scale Feature Attention Mechanism (FAM) for local feature extraction, a Local Feature Fusion (LFF) module for cross-scale feature synthesis, and a Global Feature Representation (GFR) module using Vision Transformers to capture global perceptual attributes. This synergistic framework effectively captures both fine-grained local distortions and broader global features that collectively define the visual quality of enhanced images. Furthermore, in the absence of a dedicated benchmark for enhanced natural images, we design the Natural Image Enhancement Database (NIED), a large-scale dataset consisting of 8,581 original images and 102,972 enhanced natural images generated through a wide array of traditional and deep learning-based enhancement techniques. Extensive experiments on NIED demonstrate that the proposed MLGQA model significantly outperforms current state-of-the-art BIQA methods in terms of both prediction accuracy and robustness.

Sea surface temperature (SST) is an essential indicator of global climate change and one of the most intuitive factors reflecting ocean conditions. Obtaining high-resolution SST data remains challenging due to limitations in physical imaging, and super-resolution via deep neural networks is a promising solution. Recently, Mamba-based approaches leveraging State Space Models (SSM) have demonstrated significant potential for long-range dependency modeling with linear complexity. However, their application to SST data super-resolution remains largely unexplored. To this end, we propose the Wavelet-assisted Mamba Super-Resolution (WMSR) framework for satellite-derived SST data. The WMSR includes two key components: the Low-Frequency State Space Module (LFSSM) and High-Frequency Enhancement Module (HFEM). The LFSSM uses 2D-SSM to capture global information of the input data, and the robust global modeling capabilities of SSM are exploited to preserve the critical temperature information in the low-frequency component. The HFEM employs the pixel difference convolution to match and correct the high-frequency feature, achieving accurate and clear textures. Through comprehensive experiments on three SST datasets, our WMSR demonstrated superior performance over state-of-the-art methods. Our codes and datasets will be made publicly available at https://github.com/oucailab/WMSR.

Semantic segmentation of high-resolution remote sensing images plays a crucial role in land-use monitoring and urban planning. Recent remarkable progress in deep learning-based methods makes it possible to generate satisfactory segmentation results. However, existing methods still face challenges in adapting network parameters to various land cover distributions and enhancing the interaction between spatial and frequency domain features. To address these challenges, we propose the Adaptive Frequency Enhancement Network (AFENet), which integrates two key components: the Adaptive Frequency and Spatial feature Interaction Module (AFSIM) and the Selective feature Fusion Module (SFM). AFSIM dynamically separates and modulates high- and low-frequency features according to the content of the input image. It adaptively generates two masks to separate high- and low-frequency components, therefore providing optimal details and contextual supplementary information for ground object feature representation. SFM selectively fuses global context and local detailed features to enhance the network’s representation capability. Hence, the interactions between frequency and spatial features are further enhanced. Extensive experiments on three publicly available datasets demonstrate that the proposed AFENet outperforms state-of-the-art methods. In addition, we also validate the effectiveness of AFSIM and SFM in managing diverse land cover types and complex scenarios. Our codes are available at https://github.com/oucailab/AFENet.

In the field of multisource remote sensing image classification, remarkable progress has been made by using the convolutional neural network (CNN) and Transformer. While CNNs are constrained by their local receptive fields, Transformers mitigate this issue with their global attention mechanism. However, Transformers come with the tradeoff of higher computational complexity. Recently, Mamba-based methods built upon the state space model (SSM) have shown great potential for long-range dependence modeling with linear complexity, but they have rarely been explored for multisource remote sensing image classification tasks. To address this issue, we propose the Multi-Scale Feature Fusion Mamba (MSFMamba) network, a novel framework designed for the joint classification of hyperspectral image (HSI) and light detection and ranging (LiDAR)/synthetic aperture radar (SAR) data. The MSFMamba network is composed of three key components: the Multi-Scale Spatial Mamba (MSpa-Mamba) block, the Spectral Mamba (Spe-Mamba) block, and the fusion Mamba (Fus-Mamba) block. The MSpa-Mamba block employs a multiscale strategy to reduce computational cost and alleviate feature redundancy in multiple scanning routes, ensuring efficient spatial feature modeling. The Spe-Mamba block focuses on spectral feature extraction, addressing the unique challenges of HSI data representation. Finally, the Fus-Mamba block bridges the heterogeneous gap between HSI and LiDAR/SAR data by extending the original Mamba architecture to accommodate dual inputs, enhancing cross-modal feature interactions and enabling seamless data fusion. Together, these components enable MSFMamba to effectively tackle the challenges of multisource data classification, delivering improved performance with optimized computational efficiency. Comprehensive experiments on four real-world multisource remote sensing datasets (Berlin, Augsburg, Houston2018, and Houston2013) demonstrate the superiority of MSFMamba outperforms several state-of-the-art methods and achieves overall accuracies of 76.92%, 91.38%, 92.38%, and 92.86%, respectively. The source codes of MSFMamba will be publicly available athttps://github.com/oucailab/MSFMamba.

Multi-source remote sensing data joint classification aims to provide accuracy and reliability of land cover classification by leveraging the complementary information from multiple data sources. Existing methods confront two challenges: inter-frequency multi-source feature coupling and inconsistency of complementary information exploration. To solve these issues, we present a Prototype-based Information Compensation Network (PICNet) for land cover classification based on HSI and SAR/LiDAR data. Specifically, we first design a frequency interaction module to enhance the inter-frequency coupling in multi-source feature extraction. The multi-source features are first decoupled into high- and low-frequency components. Then, these features are recoupled to achieve efficient inter-frequency communication. Afterward, we design a prototype-based information compensation module to model the global multi-source complementary information. Two sets of learnable modality prototypes are introduced to represent the global modality information of multi-source data. Subsequently, cross-modal feature integration and alignment are achieved through cross-attention computation between the modality-specific prototype vectors and the raw feature representations. Extensive experiments on three public datasets demonstrate the significant superiority of our PICNet over state-of-the-art methods. The codes are available at https://github.com/oucailab/PICNet.

Hyperspectral image (HSI) and light detection and ranging (LiDAR) data joint classification is a challenging task. Existing multisource remote sensing data classification methods often rely on human-designed frameworks for feature extraction, which heavily depend on expert knowledge. To address these limitations, we propose a novel dynamic cross-modal feature interaction network (DCMNet), the first framework leveraging a dynamic routing mechanism for HSI and LiDAR classification. Specifically, our approach introduces three feature interaction blocks: bilinear spatial attention block (BSAB), bilinear channel attention block (BCAB), and integration convolutional block (ICB). These blocks are designed to effectively enhance spatial, spectral, and discriminative feature interactions. A multilayer routing space with routing gates is designed to determine optimal computational paths, enabling data-dependent feature fusion. Additionally, bilinear attention mechanisms are employed to enhance feature interactions in spatial and channel representations. Extensive experiments on three public HSI and LiDAR datasets demonstrate the superiority of DCMNet over the state-of-the-art methods. Our codes are available athttps://github.com/oucailab/DCMNet.

Three-dimensional digital urban reconstruction from multi-view aerial images is a critical application where deep multi-view stereo (MVS) methods outperform traditional techniques. However, existing methods commonly overlook the key differences between aerial and close-range settings, such as varying depth ranges along epipolar lines and insensitive feature-matching associated with low-detailed aerial images. To address these issues, we propose an Adaptive Depth Range MVS (ADR-MVS), which integrates monocular geometric cues to improve multi-view depth estimation accuracy. The key component of ADR-MVS is the depth range predictor, which generates adaptive range maps from depth and normal estimates using cross-attention discrepancy learning. In the first stage, the range map derived from monocular cues breaks through predefined depth boundaries, improving feature-matching discriminability and mitigating convergence to local optima. In later stages, the inferred range maps are progressively narrowed, ultimately aligning with the cascaded MVS framework for precise depth regression. Moreover, a normal-guided cost aggregation operation is specially devised for aerial stereo images to improve geometric awareness within the cost volume. Finally, we introduce a normal-guided depth refinement module that surpasses existing RGB-guided techniques. Experimental results demonstrate that ADR-MVS achieves state-of-the-art performance on the WHU, LuoJia-MVS, and München datasets, while exhibits superior computational complexity.

Accurately forecasting sea ice concentration (SIC) in the Arctic is critical to global ecosystem health and navigation safety. However, current methods still is confronted with two challenges: 1) these methods rarely explore the long-term feature dependencies in the frequency domain. 2) they can hardly preserve the high-frequency details, and the changes in the marginal area of the sea ice cannot be accurately captured. To this end, we present a Frequency-Compensated Network (FCNet) for Arctic SIC prediction on a daily basis. In particular, we design a dual-branch network, including branches for frequency feature extraction and convolutional feature extraction. For frequency feature extraction, we design an adaptive frequency filter block, which integrates trainable layers with Fourier-based filters. By adding frequency features, the FCNet can achieve refined prediction of edges and details. For convolutional feature extraction, we propose a high-frequency enhancement block to separate high and low-frequency information. Moreover, high-frequency features are enhanced via channel-wise attention, and temporal attention unit is employed for low-frequency feature extraction to capture long-range sea ice changes. Extensive experiments are conducted on a satellite-derived daily SIC dataset, and the results verify the effectiveness of the proposed FCNet. Our codes and data will be made public available at: https://github.com/oucailab/FCNet.

Arctic Sea Ice Concentration (SIC) is the ratio of ice-covered area to the total sea area of the Arctic Ocean, which is a key indicator for maritime activities. Nowadays, we often use passive microwave images to display SIC, but it has low spatial resolution, and most of the existing super-resolution methods of Arctic SIC don’t take the integration of spatial and channel features into account and can’t effectively integrate the multi-scale feature. To overcome the aforementioned issues, we propose MFM-Net for Arctic SIC super-resolution, which concurrently aggregates multi-scale information while integrating spatial and channel features. Extensive experiments on Arctic SIC dataset from the AMSR-E/AMSR-2 SIC DT-ASI products from Ocean University of China validate the effectiveness of porposed MFM-Net.

In this paper, we proposed large selective kernel and sparse attention network (LSKSANet) for remote sensing image semantic segmentation. The LSKSANet is a lightweight network that effectively combines convolution with sparse attention mechanisms. Specifically, we design large selective kernel module to decomposing the large kernel into a series of depth-wise convolutions with progressively increasing dilation rates, thereby expanding the receptive field without significantly increasing the computational burden. In addition, we introduce the sparse attention to keep the most useful selfattention values for better feature aggregation. Experimental results on the Vaihingen and Postdam datasets demonstrate the superior performance of the proposed LSKSANet over state-of-the-art methods.

Multi-source remote sensing data classification has emerged as a prominent research topic with the advancement of various sensors. Existing multi-source data classification methods are susceptible to irrelevant information interference during multi-source feature extraction and fusion. To solve this issue, we propose a sparse focus network for multi-source data classification. Sparse attention is employed in Transformer block for HSI and SAR/LiDAR feature extraction, thereby the most useful self-attention values are maintained for better feature aggregation. Furthermore, cross-attention is used to enhance multi-source feature interactions, and further improves the efficiency of cross-modal feature fusion. Experimental results on the Berlin and Houston2018 datasets highlight the effectiveness of SF-Net, outperforming existing state-of-the-art methods.

Although recent masked image modeling (MIM)-based HSI-LiDAR/SAR classification methods have gradually recognized the importance of the spectral information, they have not adequately addressed the redundancy among different spectra, resulting in information leakage during the pretraining stage. This issue directly impairs the representation ability of the model. To tackle the problem, we propose a new strategy, named Mining Redundant Spectra (MRS). Unlike randomly masking spectral bands, MRS selectively masks them by similarity to increase the reconstruction difficulty. Specifically, a random spectral band is chosen during pretraining, and the selected and highly similar bands are masked. Experimental results demonstrate that employing the MRS strategy during the pretraining stage effectively improves the accuracy of existing MIM-based methods on the Berlin and Houston 2018 datasets.

Hyperspectral image (HSI) contains abundant spatial and spectral information, making it highly valuable for unmixing. In this paper, we propose a Dual-Stream Attention Network (DSANet) for HSI unmixing. The endmembers and abundance of a pixel in HSI have high correlations with its adjacent pixels. Therefore, we adopt a “many to one” strategy to estimate the abundance of the central pixel. In addition, we adopt multiview spectral method, dividing spectral bands into multiple partitions with low correlations to estimate abundances. To aggregate the estimated abundances for complementary from the two branches, we design a cross-fusion attention network to enhance valuable information. Extensive experiments have been conducted on two real datasets, which demonstrate the effectiveness of our DSANet.

Hyperspectral image (HSI) denoising is critical for the effective analysis and interpretation of hyperspectral data. However, simultaneously modeling global and local features is rarely explored to enhance HSI denoising. In this letter, we propose a hybrid convolution and attention network (HCANet), which leverages both the strengths of convolution neural networks (CNNs) and Transformers. To enhance the modeling of both global and local features, we have devised a convolution and attention fusion module aimed at capturing long-range dependencies and neighborhood spectral correlations. Furthermore, to improve multi-scale information aggregation, we design a multi-scale feed-forward network to enhance denoising performance by extracting features at different scales. Experimental results on mainstream HSI datasets demonstrate the rationality and effectiveness of the proposed HCANet. The proposed model is effective in removing various types of complex noise. Our codes are available at https://github.com/summitgao/HCANet.

Hyperspectral image (HSI) and synthetic aperture radar (SAR) data joint classification is a crucial and yet challenging task in the field of remote sensing image interpretation. However, feature modeling in the existing methods is deficient to exploit the abundant global, spectral, and local features simultaneously, leading to suboptimal classification performance. To solve the problem, we propose a hierarchical attention and parallel filter fusion network for multisource data classification. Concretely, we design a hierarchical attention module (HAM) for hyperspectral feature extraction. This module integrates global, spectral, and local features simultaneously to provide more comprehensive feature representation. In addition, we develop parallel filter fusion module (PFFM), which enhances cross-modal feature interactions among different spatial locations in the frequency domain. Extensive experiments on two multisource remote sensing data classification datasets verify the superiority of our proposed method over current state-of-the-art classification approaches. Specifically, our proposed method achieves 91.44% and 80.51% of overall accuracy (OA) on the respective datasets, highlighting its superior performance.

Deep-learning (DL)-based methods have been increasingly used in hyperspectral unmixing (HU), especially the recent trend of unsupervised autoencoder (AE) networks, which have achieved excellent performances. Although some existing unmixing methods take spatial information into account, the utilization of spatial structure is not sufficient and effective. In this letter, we present a deep attention-guided spatial–spectral network for hyperspectral image (HSI) unmixing called DASS-Net, which adopts a parallel dual-stream structure. We design a neighborhood spatial attention (NSA) module, where the abundance features of the central pixel are dynamically weighted by the coarse-grained features of the neighborhood pixels. In addition, a dual-gated mechanism is introduced to further integrate and express the spatial and spectral information. Experimental results show that the proposed DASS-Net performs particularly well in endmember extraction and outperforms all compared methods.

Synthetic aperture radar (SAR) image change detection is critical in remote sensing image analysis. Recently, the attention mechanism has been widely used in change detection tasks. However, existing attention mechanisms often use downsampling operations such as average pooling on the key and value components to enhance computational efficiency. These irreversible operations result in the loss of high-frequency components and other important information. To address this limitation, we develop wavelet-based bi-dimensional aggregation network (WBANet) for SAR image change detection. We design a wavelet-based self-attention block that includes discrete wavelet transform (DWT) and inverse DWT (IDWT) operations on key and value components. Hence, the feature undergoes downsampling without any loss of information, while simultaneously enhancing local contextual awareness through an expanded receptive field. In addition, we have incorporated a bi-dimensional aggregation module (BAM) that boosts the nonlinear representation capability by merging spatial and channel information via broadcast mechanism. Experimental results on three SAR datasets demonstrate that our WBANet significantly outperforms contemporary state-of-the-art methods. Specifically, our WBANet achieves 98.33%, 96.65%, and 96.62% of percentage of correct classification (PCC) on the respective datasets, highlighting its superior performance. Source codes are available athttps://github.com/summitgao/WBANet.

Currently, high-precision fully supervised methods are mainly used in the change detection tasks of high-resolution remote sensing images, which require a large amount of labeled data. However, labeling huge amounts of sample data is costly. In order to solve this limitation, a SemiPSENet based on consistent regularization is proposed. SemiPSENet consists of two parts: a supervised phase and an unsupervised phase, which is trained jointly with a small amount of labeled data and a large amount of unlabeled data. In the supervision phase, to generate a multiscale feature map that can fuse more context information, a GELU_PSP module is designed, which can greatly reduce the loss of semantic information. The squeeze-and-excitation (SE) attention module is added after GELU_PSP module, which can highlight important changing features and neglect irrelevant information based on the importance of each feature channel. In the unsupervised phase, to process the unlabeled data, the consistency regularization method is adopted, which can make the predictive change map have the consistency ability under different random perturbations, so that the semantic information can fully be used in the unlabeled image. Experiments on two public datasets LEVIR-CD and WHU show that the proposed SemiPSENet can achieve excellent detection results by relying on the training of a small amount of labeled data and a large amount of unlabeled data. Notably, our network performs better than other SOTA methods when using the same amount of labeled data.

Recently, the methods that combine the merits of the numerical model and the deep learning to improve the prediction accuracy of the sea surface temperature (SST) have received considerable attention. Existing methods usually apply the output of the numerical model as the physical knowledge to guide the training of the deep learning models. However, the physical knowledge in the observed data has not been fully exploited. With the development of observational instruments and techniques, an increasing amount of observational data has been collected. These data can be utilized for the exploration of physical knowledge. Toward this end, we propose novel scheme for SST prediction, which applies generative adversarial networks (GANs) to analyze the physical knowledge in the historical data. In particular, two GAN models are trained with numerical model data and observed data separately. Afterward, the physical knowledge is extracted from the observed data which is not contained in the data generated by the numerical model by comparing the learned physical feature from the two pre-trained GAN models. Finally, to validate the relevance of the physical knowledge which we have discovered, the extracted features are added into the numerical model data which are called newly corrected data. Besides, we train two spatial-temporal models over the newly corrected dataset and the original numerical model data for SST prediction, respectively. The experimental results show that the newly corrected dataset performs better than using the original numerical model for SST prediction.

In conventional remote sensing change detection (RSCD) procedures, extensive manual labeling for bi-temporal images is first required to maintain the performance of subsequent fully supervised training. However, pixel-level labeling for change detection (CD) tasks is very complex and time-consuming. In this article, we explore a novel self-supervised contrastive framework applicable to the RSCD task, which promotes the model to accurately capture spatial, structural, and semantic information through the domain adapter (DA) and the hierarchical contrastive head. The proposed SSLChange framework accomplishes self-learning only by taking a single-temporal sample and can be flexibly transferred to mainstream CD baselines. With self-supervised contrastive learning, feature representation pretraining can be performed directly based on the original data even without labeling. After a certain number of labels are subsequently obtained, the pretrained features will be aligned with the labels for fully supervised fine-tuning. Without introducing any additional data or labels, the performance of downstream baselines will experience a significant enhancement. Experimental results on two entire datasets and six diluted datasets show that our proposed SSLChange improves the performance and stability of CD baseline in data-limited situations. The code of SSLChange is available athttps://github.com/MarsZhaoYT/SSLChange

Removing noise from hyperspectral images (HSIs) has been widely regarded as one of the most meaningful preprocessing tasks in remote sensing image interpretation. In this paper, we aim to extend the Transformer backbone to HSI denoising, and propose a Multi-scale Transformer Denoising Network (MTDNet). Specifically, we design a multi-head global attention module to alleviate the computational burden caused by self-attention. Furthermore, we propose a multi-scale feed-forward network in which three branches of multi-scale features are extracted through dilated convolution. It enriches the non-linear feature transformation in the Transformer block. Both the objective and subjective experiments on the ICVL dataset demonstrate the superiority of the proposed MTDNet over four closely related methods.

Most of existing mutli-source remote sensing data classification methods are based on convolutional neural networks. Recently, the emergence of Vision Transformer greatly challenges the dominance of CNN-based methods. The self-attention mechanism in Transformer and other dynamic networks imply that high-order feature interactions are beneficial to improve the feature representation and fusion. To explore the high-order feature interactions in multi-source image fusion, in this paper, we proposed a novel recursive feature interactive fusion network. It is composed of cross-shaped window self-attention encoder, and recursive feature interactive fusion. We use gated convolution recursively to mix multi-modal features and exploit their spatial relations. Experimental results on two datasets show that the proposed method achieves better performance than closely related methods.

Existing hyperspectral image (HSI) and LiDAR data joint classification methods commonly treat LiDAR data equally with HSI in the network. As a result, these methods may fail to effectively leverage the spectral features from HSI and elevation information from LiDAR. In this paper, we show that better cross-modal alignments can be achieved through an HSI encoder for jointly embedding elevation features from LiDAR during spectral feature encoding. To this end, we propose a Gated-Cross Aggregation Network (GCA-Net) to fully investigate the complementary clues hidden in multi-source data progressively. The beneficial spectral-elevation cues are then exploited by cross-attention feature fusion. Then, useful LiDAR features are integrated into the HSI features via an elevation gating module, which occurs at each stage of the network. Experimental results on the Houston 2013 dataset and Trento dataset reveal that the proposed GCA-Net achieves better performance than several closely related methods.

Existing hyperspectral image unmixing methods acquire information from single-view, and therefore can hardly make full use of diverse spectral information, and the learning of space and spectrum is often relatively independent and cannot be combined effectively. In order to solve the problem of insufficient feature representation caused by the single-view spectral information and the lack of close relationship between spatial and spectral learning, we introduce the idea of multiview data construction, which divides the spectral bands into different views for multiview learning. In addition, we proposed a spatial-spectral siamese network. Deep collaborative learning is used to construct an unmixing model by combining multiview representation and the siamese network. Experimental results on the Jasper Ridge and Urban datasets demonstrate the effectiveness of the proposed method.

DEtection TRansformer (DETR) and its variants (DETRs) achieved impressive performance in general object detection. However, in crowded pedestrian detection, the performance of DETRs is still unsatisfactory due to the inappropriate sample selection method which results in more false positives. To settle the issue, we propose a simple but effective sample selection method for DETRs, Sample Selection for Crowded Pedestrians (SSCP), which consists of the constraint-guided label assignment scheme (CGLA) and the utilizability-aware focal loss (UAFL). Our core idea is to select learnable samples for DETRs and adaptively regulate the loss weights of samples based on their utilizability. Specifically, in CGLA, we proposed a new cost function to ensure that only learnable positive training samples are retained and the rest are negative training samples. Further, considering the utilizability of samples, we designed UAFL to adaptively assign different loss weights to learnable positive samples depending on their gradient ratio and IoU. Experimental results show that the proposed SSCP effectively improves the baselines without introducing any overhead in inference. Especially, Iter Deformable DETR is improved to 39.7(-2.0)% MR on Crowdhuman and 31.8(-0.4)% MR on Citypersons.

Abstract The appearance of specular highlights in images is one main factor affecting accurate material or object recognition tasks. Such an appearance has a misleading effect on the true gradient information found in transmissive material images. Certain methods use specular highlights as an intrinsic feature of transparency to detect transparent objects. However, this process reduces the robustness of methods in applications with opaque and shiny materials and in the classification of tasks among related features, such as transparency and translucency. Thus, correcting this artefact can enhance texture‐ or gradient‐based image and video analyses. However, the correction of small or large regions with specular highlights from transmissive materials, such as glass, plastic and water, is complex and ambiguous. These materials are sensitive to specular highlights and exhibit high degrees of reflection. In this study, we propose a deep learning framework to address the problem. A partial convolution‐based inpainting method is integrated with automatic semantic mask generation by using a simple adaptive binarization to detect highlight spots during training and inference. The proposed framework improves the learning process by capturing the semantic nature of specular highlights. Moreover, the framework eliminates the use of image‐mask pairs during inference and avoids predefined irregular random mask training. We qualitatively and quantitatively evaluate the proposed framework by using new and existing publicly available datasets that contain specular images. Experimental results show that our framework registers competitive performance and considerably reduces computational time.

Hyperspectral unmixing is a critical yet challenging task in hyperspectral image interpretation.Recently, great efforts have been made to solve the hyperspectral unmixing task via deep autoencoders.However, existing networks mainly focus on extracting spectral features from mixed pixels, and the employment of spatial feature prior knowledge is still insufficient.To this end, we put forward a spatial attention weighted unmixing network, dubbed as SAWU-Net, which learns a spatial attention network and a weighted unmixing network in an end-to-end manner for better spatial feature exploitation.In particular, we design a spatial attention module, which consists of a pixel attention block and a window attention block to efficiently model pixelbased spectral information and patch-based spatial information, respectively.While in the weighted unmixing framework, the central pixel abundance is dynamically weighted by the coarsegrained abundances of surrounding pixels.In addition, SAWU-Net generates dynamically adaptive spatial weights through the spatial attention mechanism, so as to dynamically integrate surrounding pixels more effectively.Experimental results on real and synthetic datasets demonstrate the better accuracy and superiority of SAWU-Net, which reflects the effectiveness of the proposed spatial attention mechanism.

Synthetic aperture radar (SAR) image change detection is a critical task and has received increasing attentions in the remote sensing community. However, existing SAR change detection methods are mainly based on convolutional neural networks (CNNs), with limited consideration of global attention mechanism. In this letter, we explore Transformer-like architecture for SAR change detection to incorporate global attention. To this end, we propose a convolution and attention mixer (CAMixer). First, to compensate the inductive bias for Transformer, we combine self-attention with shift convolution in a parallel way. The parallel design effectively captures the global semantic information via the self-attention and performs local feature extraction through shift convolution simultaneously. Second, we adopt a gating mechanism in the feed-forward network to enhance the non-linear feature transformation. The gating mechanism is formulated as the element-wise multiplication of two parallel linear layers. Important features can be highlighted, leading to high-quality representations against speckle noise. Extensive experiments conducted on three SAR datasets verify the superior performance of the proposed CAMixer. The source codes will be publicly available at https://github.com/summitgao/CAMixer.

Masked image modeling (MIM) is a highly popular and effective self-supervised learning method for image understanding. Existing MIM-based methods mostly focus on spatial feature modeling, neglecting spectral feature modeling. Meanwhile, existing MIM-based methods use Transformer for feature extraction, some local or high-frequency information may get lost. To this end, we propose a spatial-spectral masked auto-encoder (SS-MAE) for HSI and LiDAR/SAR data joint classification. Specifically, SS-MAE consists of a spatial-wise branch and a spectral-wise branch. The spatial-wise branch masks random patches and reconstructs missing pixels, while the spectral-wise branch masks random spectral channels and reconstructs missing channels. Our SS-MAE fully exploits the spatial and spectral representations of the input data. Furthermore, to complement local features in the training stage, we add two lightweight CNNs for feature extraction. Both global and local features are taken into account for feature modeling. To demonstrate the effectiveness of the proposed SS-MAE, we conduct extensive experiments on three publicly available datasets. Extensive experiments on three multi-source datasets verify the superiority of our SS-MAE compared with several state-of-the-art baselines. The source codes are available at https://github.com/summitgao/SS-MAE.

Traditionally, numerical models have been deployed in oceanography studies to simulate ocean dynamics by representing physical equations. However, many factors pertaining to ocean dynamics seem to be ill-defined. We argue that transferring physical knowledge from observed data could further improve the accuracy of numerical models when predicting Sea Surface Temperature (SST). Recently, the advances in earth observation technologies have yielded a monumental growth of data. Consequently, it is imperative to explore ways in which to improve and supplement numerical models utilizing the ever-increasing amounts of historical observational data. To this end, we introduce a method for SST prediction that transfers physical knowledge from historical observations to numerical models. Specifically, we use a combination of an encoder and a generative adversarial network (GAN) to capture physical knowledge from the observed data. The numerical model data is then fed into the pre-trained model to generate physics-enhanced data, which can then be used for SST prediction. Experimental results demonstrate that the proposed method considerably enhances SST prediction performance when compared to several state-of-the-art baselines.

Linear spectral unmixing is an important technique in the analysis of mixed pixels in hyperspectral images. In recent years, deep learning-based methods have been garnering increasing attention in hyperspectral unmixing; especially, unsupervised autoencoder (AE) networks that have achieved excellent unmixing performance are a recent trend. While most approaches use spatial information, it is well known that hyperspectral data are characterized by a large number of narrow spectral bands. In order to take full advantage of the hyperspectral bands in unmixing and the spatial information, in this article, we explore multiview spectral and spatial information in an AE-based unmixing framework. We introduce multiview spectral information through spectral partitioning and propose a multiview spatial–spectral two-stream network, called MSSS-Net, which simultaneously learns a spatial stream network and a multiview spectral stream network in an end-to-end fashion for more efficient unmixing. The MSSS-Net is a two-stream deep unmixing network sharing a decoder, where its two AE networks employ recurrent neural networks (RNNs) to collaboratively utilize multiview spectral and spatial information. The spatial stream network branch extracts the spatial features of pixels and its neighbors, while the multiview spectral stream network branch exploits the multiview spectral bands of a pixel. Meanwhile, we design a cascaded bidirectional and unidirectional RNNs’ encoder structure for multiview spatial–spectral information to learn more discriminative deep patch-pixel features. Extensive ablation studies and experiments on both synthetic and real datasets demonstrate the superiority of the MSSS-Net over state-of-the-art unmixing methods.

The joint hyperspectral image (HSI) and light detection and ranging (LiDAR) data classification aims to interpret ground objects at more detailed and precise level. Although deep learning methods have shown remarkable success in the multisource data classification task, self-supervised learning has rarely been explored. It is commonly nontrivial to build a robust self-supervised learning model for multisource data classification, due to the fact that the semantic similarities of neighborhood regions are not exploited in the existing contrastive learning framework. Furthermore, the heterogeneous gap induced by the inconsistent distribution of multisource data impedes the classification performance. To overcome these disadvantages, we propose a nearest neighbor-based contrastive learning network (NNCNet), which takes full advantage of large amounts of unlabeled data to learn discriminative feature representations. Specifically, we propose a nearest neighbor-based data augmentation scheme to use enhanced semantic relationships among nearby regions. The intermodal semantic alignments can be captured more accurately. In addition, we design a bilinear attention module to exploit the second-order and even high-order feature interactions between the HSI and LiDAR data. Extensive experiments on four public datasets demonstrate the superiority of our NNCNet over state-of-the-art methods. The source codes are available athttps://github.com/summitgao/NNCNet.

Sea subsurface temperature, an essential component of aquatic wildlife, underwater dynamics, and heat transfer with the sea surface, is affected by global warming in climate change. Existing research is commonly based on either physics-based numerical models or data-based models. Physical modeling and machine learning are traditionally considered as two unrelated fields for the sea subsurface temperature prediction task, with very different scientific paradigms (physics-driven and data-driven). However, we believe that both methods are complementary to each other. Physical modeling methods can offer the potential for extrapolation beyond observational conditions, while data-driven methods are flexible in adapting to data and are capable of detecting unexpected patterns. The combination of both approaches is very attractive and offers potential performance improvement. In this article, we propose a novel framework based on a generative adversarial network (GAN) combined with a numerical model to predict sea subsurface temperature. First, a GAN-based model is used to learn the simplified physics between the surface temperature and the target subsurface temperature in the numerical model. Then, observation data are used to calibrate the GAN-based model parameters to obtain a better prediction. We evaluate the proposed framework by predicting daily sea subsurface temperature in the South China Sea. Extensive experiments demonstrate the effectiveness of the proposed framework compared to existing state-of-the-art methods.

Hyperspectral images (HSIs) are widely used in agriculture and environmental monitoring. However, due to the climate and weather conditions, the acquired images commonly contain noise or detail loss, which greatly affects the image interpretation. To solve the problem, we propose a HSI denoising network based on the combination of Transformer and convolutional neural network (CNN), which can effectively improve the complex noise while ensuring image quality by using the powerful local analysis ability of CNN and the effective global feature interaction of Transformer. Extensive experiments on HSI dataset show that the proposed method outperforms four closely related methods.

Self-attention mechanism has been widely used in deep neural networks to improve the feature representation. In this paper, the self-attention mechanism is employed to solve the SAR change detection task. Specifically, we propose a Multi-scale Attention Convolution Network, MACNet for short, which exploits the multiscale features by a linear attention weight module. To enrich the feature representation space, we extract the spatial information of feature maps in a multi-branch way. Furthermore, we design a linear attention weight module, which emphasizes the important channels adaptively and fuses the contextual information from different scales. Experiments are conducted on three SAR datasets, and the results demonstrate the superior performance of the proposed method.

To exploit the multi-scale information to improve the feature representation, we propose a multi-scale feature fusion network for hyperspectral image (HSI) and LiDAR data joint classification. The model is comprised of three parts: feature extraction module, feature fusion module, and MSF(Multi-Scale Fusion) module. The feature fusion module integrates the multi-modal information into the two inputs by fusing attention. MSF module integrates richer semantic information by integrating multi-scale information to improve performance. Experimental results show that the proposed method is effective in multi-modal data joint classification.

The goal of General Continual Learning (GCL) is to preserve learned knowledge and learn new knowledge with constant memory from an infinite data stream where task boundaries are blurry. Distilling the model's response of reserved samples between the old and the new models is an effective way to achieve promise performance on GCL. However, it accumulates the inherent old model's response bias and is not robust to model changes. To this end, we propose an Online Contrastive Distillation Network (OCD-Net) to tackle these problems, which explores the merit of the student model in each time step to guide the training process of the student model. Concretely, the teacher model is devised to help the student model to consolidate the learned knowledge, which is trained online via integrating the model weights of the student model to accumulate the new knowledge. Moreover, our OCD-Net incorporates both relation and adaptive response to help the student model alleviate the catastrophic forgetting, which is also beneficial for the teacher model preserves the learned knowledge. Extensive experiments on six benchmark datasets demonstrate that our proposed OCD-Net significantly outperforms state-of-the-art approaches in 3.26%~8.71% with various buffer sizes. Our code is available at https://github.com/lijincm/OCD-Net.

Photometric stereo aims to reconstruct 3D geometry by recovering the dense surface orientation of a 3D object from multiple images under differing illumination. Traditional methods normally adopt simplified reflectance models to make the surface orientation computable. However, the real reflectances of surfaces greatly limit applicability of such methods to real-world objects. While deep neural networks have been employed to handle non-Lambertian surfaces, these methods are subject to blurring and errors, especially in high-frequency regions (such as crinkles and edges), caused by spectral bias: neural networks favor low-frequency representations so exhibit a bias towards smooth functions. In this paper, therefore, we propose a self-learning conditional network with multi-scale features for photometric stereo, avoiding blurred reconstruction in such regions. Our explorations include: (i) a multi-scale feature fusion architecture, which keeps high-resolution representations and deep feature extraction, simultaneously, and (ii) an improved gradient-motivated conditionally parameterized convolution (GM-CondConv) in our photometric stereo network, with different combinations of convolution kernels for varying surfaces. Extensive experiments on public benchmark datasets show that our calibrated photometric stereo method outperforms the state-of-the-art.

Recently, change detection methods for synthetic aperture radar (SAR) images based on convolutional neural networks (CNN) have gained increasing research attention. However, existing CNN-based methods neglect the interactions among multilayer convolutions, and errors involved in the preclassification restrict the network optimization. To this end, we proposed a layer attention-based noise-tolerant network, termed LANTNet. In particular, we design a layer attention module that adaptively weights the feature of different convolution layers. In addition, we design a noise-tolerant loss function that effectively suppresses the impact of noisy labels. Therefore, the model is insensitive to noisy labels in the preclassification results. The experimental results on three SAR datasets show that the proposed LANTNet performs better compared to several state-of-the-art methods. The source codes are available at https://github.com/summitgao/LANTNet.

Change detection from synthetic aperture radar (SAR) imagery is a critical yet challenging task. Existing methods mainly focus on feature extraction in the spatial domain, and little attention has been paid to the frequency domain. Furthermore, in patch-wise feature analysis, some noisy features in the marginal region may be introduced. To tackle the above two challenges, we propose a dual-domain network (DDNet). Specifically, we take features from the discrete cosine transform (DCT) domain into consideration and the reshaped DCT coefficients are integrated into the proposed model as the frequency domain branch. Feature representations from both frequency and spatial domain are exploited to alleviate the speckle noise. In addition, we further propose a multi-region convolution (MRC) module, which emphasizes the central region of each patch. The contextual information and central region features are modeled adaptively. The experimental results on three SAR data sets demonstrate the effectiveness of the proposed model. Our codes are available athttps://github.com/summitgao/SAR_CD_DDNet.

Existing remote sensing change detection methods are heavily affected by seasonal variation. Since vegetation colors are different between winter and summer, such variations are inclined to be falsely detected as changes. In this letter, we proposed an image translation method to solve the problem. A style-based recalibration module is introduced to capture seasonal features effectively. Then, a new style discriminator is designed to improve the translation performance. The discriminator can not only produce a decision for the fake or real sample but also return a style vector according to the channel-wise correlations. Extensive experiments are conducted on the season-varying data set. The experimental results show that the proposed method can effectively perform image translation, thereby consistently improving the season-varying image change detection performance. Our codes and data are available athttps://github.com/summitgao/RSIT_SRM_ISD.

Remote sensing image captioning (RSIC) has great significance in image understanding, which describes the image content in natural language. Existing methods are mainly based on deep learning and rely on the encoder–decoder model to generate sentences. In the decoding process, recurrent neural network (RNN) and long short-term memory (LSTM) are normally applied to sequentially generate image captions. In this letter, the transformer encoder–decoder is combined with grid features to improve the RSIC performance. First, the pretrained convolutional neural network (CNN) is used to extract grid-based visual features, which are encoded as vectorial representations. Then, the transformer outputs semantic descriptions to bridge visual features and natural language. Besides, the self-critical sequence training (SCST) strategy is applied to further optimize the image captioning model and improve the quality of generated sentences. Extensive experiments are organized on three public datasets of RSCID, UCM-Captions, and Sydney-Captions. Experimental results demonstrate the effectiveness of SCST strategy and the proposed method achieves superior performance compared with the state-of-the-art image captioning approaches on the RSCID dataset.

Clustering is one of the fundamental tasks in computer vision and pattern recognition. Recently, deep clustering methods (algorithms based on deep learning) have attracted wide attention with their impressive performance. Most of these algorithms combine deep unsupervised representation learning and standard clustering together. However, the separation of representation learning and clustering will lead to suboptimal solutions because the two-stage strategy prevents representation learning from adapting to subsequent tasks (e.g., clustering according to specific cues). To overcome this issue, efforts have been made in the dynamic adaption of representation and cluster assignment, whereas current state-of-the-art methods suffer from heuristically constructed objectives with the representation and cluster assignment alternatively optimized. To further standardize the clustering problem, we audaciously formulate the objective of clustering as finding a precise feature as the cue for cluster assignment. Based on this, we propose a general-purpose deep clustering framework, which radically integrates representation learning and clustering into a single pipeline for the first time. The proposed framework exploits the powerful ability of recently developed generative models for learning intrinsic features, and imposes an entropy minimization on the distribution of the cluster assignment by a dedicated variational algorithm. The experimental results show that the performance of the proposed method is superior, or at least comparable to, the state-of-the-art methods on the handwritten digit recognition, fashion recognition, face recognition, and object recognition benchmark datasets.

Convolutional long short-term memory (ConvLSTM) has received much attention for hyperspectral image (HSI) classification due to its ability of modeling long-range correlations, which, however, is vulnerable to too many parameters and insufficient training, limiting its classification accuracy, especially for small samples. Different from it, traditional hand-crafted methods extract the features with basic attributes of HSIs, which can provide the lack of details and interpretability of deep semantic features. However, existing methods fail to incorporate their complementarity for HSI classification. As such, a Pseudo complex-valued (CV) Deformable ConvLSTM Neural Network with mutual Attention learning (APDCLNN) is proposed, providing a new way to realize the collaborative learning of hand-crafted and deep features for HSI classification. First, a 2-D pseudo CV deformable ConvLSTM (PDConvLSTM2D) cell is designed using deformable convolution and complex operations, with which a spatial–spectral PDConvLSTM2D neural network (SSPDCL2DNN) is built to extract scale- and spectral-enhanced deep spatial–spectral features. Then, 3-D Gabor filter is used to extract hand-crafted features, and a mutual attention-based multimodality feature learning and fusion (MAMLF) module is designed to integrate them into deep features for training and optimization of SSPDCL2DNN. Finally, an attention loss subnetwork is designed to refine the classification results. As we know, this is the first attempt to apply the idea of mutual attention learning to fuse hand-crafted and deep features for HSI classification. Extensive experiments on three widely used HSI datasets show the advantages of our model over other deep methods in terms of both quantitative and visual quality.

Linear spectral unmixing is an essential technique in hyperspectral image (HSI) processing and interpretation. In recent years, deep learning-based approaches have shown great promise in hyperspectral unmixing (HU), in particular, unsupervised unmixing methods based on autoencoder (AE) networks are a recent trend. The AE model, which automatically learns low-dimensional representations (abundances) and reconstructs data with their corresponding bases (endmembers), has achieved superior performance in HU. In this article, we explore the effective utilization of spatial and spectral information in AE-based unmixing networks. Important findings on the use of spatial and spectral information in the AE framework are discussed. Inspired by these findings, we propose a spatial–spectral collaborative unmixing network, called SSCU-Net, which learns a spatial AE network and a spectral AE network in an end-to-end manner to more effectively improve the unmixing performance. SSCU-Net is a two-stream deep network and shares an alternating architecture, where the two AE networks are efficiently trained in a collaborative way for estimation of endmembers and abundances. Meanwhile, we propose a new spatial AE network by introducing a superpixel segmentation method based on abundance information, which greatly facilitates the employment of spatial information and improves the accuracy of unmixing network. Moreover, extensive ablation studies are carried out to investigate the performance gain of SSCU-Net. Experimental results on both synthetic and real hyperspectral datasets illustrate the effectiveness and competitiveness of the proposed SSCU-Net compared with several state-of-the-art HU methods.

Existing deep-network based texture synthesis approaches all focus on fine-grained control of texture generation by synthesizing images from exemplars. Since the networks employed by most of these methods are always tied to individual exemplar textures, a large number of individual networks have to be trained when modeling various textures. In this paper, we propose to generate textures directly from coarse-grained control or high-level guidance, such as texture categories, perceptual attributes and semantic descriptions. We fulfill the task by parsing the generation process of a texture into the three-level Bayesian hierarchical model. A coarse-grained signal first determines a distribution over Markov random fields. Then a Markov random field is used to model the distribution of the final output textures. Finally, an output texture is generated from the sampled Markov random field distribution. At the bottom level of the Bayesian hierarchy, the isotropic and ergodic characteristics of the textures favor a construction that consists of a fully convolutional network. The proposed method integrates texture creation and texture synthesis into one pipeline for real-time texture generation, and enables users to readily obtain diverse textures with arbitrary scales from high-level guidance only. Extensive experiments demonstrate that the proposed method is capable of generating plausible textures that are faithful to user-defined control, and achieving impressive texture metamorphosis by interpolation in the learned texture manifold.

In this paper, an efficient linear self-attention fusion model is proposed for the task of hyperspectral image (HSI) and LiDAR data joint classification. The proposed method is comprised of a feature extraction module, an attention module, and a fusion module. The attention module is a plug-and-play linear self-attention module that can be extensively used in any model. The proposed model has achieved the overall accuracy of 95.40% on the Houston dataset. The experimental results demonstrate the superiority of the proposed method over other state-of-the-art models.

Hyperspectral images (HSIs) have been widely applied in many fields, such as military, agriculture, and environment monitoring. Nevertheless, HSIs commonly suffer from various types of noise during acquisition. Therefore, denoising is critical for HSI analysis and applications. In this paper, we propose a novel blind denoising method for HSIs based on Multi-Stream Denoising Network (MSDNet). Our network consists of the noise estimation subnetwork and denoising subnetwork. In the noise estimation subnetwork, a multiscale fusion module is designed to capture the noise from different scales. Then, the denoising subnetwork is utilized to obtain the final denoising image. The proposed MSDNet can obtain robust noise level estimation, which is capable of improving the performance of HSI denoising. Extensive experiments on HSI dataset demonstrate that the proposed method outperforms four closely related methods.

As a critical step for Polarimetric Synthetic Aperture Radar (PolSAR) images interpretation, PolSAR classification have attracted growing interests in the field of remote sensing. Recently, many novel ideas and models based on deep learning have emerged to solve the task of PolSAR image classification. The encode-decode network structure of Fully Convolutional Network (FCN) is proved to be effective for this task. However, due to the inherent disadvantages of FCN and the complex high-dimensional feature representation of PolSAR images, there are still some issues need to be addressed. Aiming at the problem that the edge of adjacent regions is not enough finely classified and the regional consistency of the same object class is relatively weak, we propose a novel PolSAR image classification method called DBFCN, which combines a well-designed edge-aware network and the improved FCN. The experimental results verify that it can effectively improve the classification accuracy of PolSAR images.

As the ground objects become increasingly complex, the classification results obtained by single source remote sensing data can hardly meet the application requirements. In order to tackle this limitation, we propose a simple yet effective attention fusion model based on Disentangled Non-local (DNL) network for hyperspectral and LiDAR data joint classification task. In this model, according to the spectral and spatial characteristics of HSI and LiDAR, a multiscale module and a convolutional neural network (CNN) are used to capture the spectral and spatial characteristics respectively. In addition, the extracted HSI and LiDAR features are fused through some operations to obtain the feature information more in line with the real situation. Finally, the above three data are fed into different branches of the DNL module, respectively. Extensive experiments on Houston dataset show that the proposed network is superior and more effective compared to several of the most advanced baselines in HSI and LiDAR joint classification missions.

Traditional synthetic-aperture radar (SAR) image change detection methods based on convolutional neural networks (CNNs) face the challenges of speckle noise and deformation sensitivity. To mitigate these issues, we proposed a multiscale capsule network (Ms-CapsNet) to extract the discriminative information between the changed and unchanged pixels. On the one hand, the multiscale capsule module is employed to exploit the spatial relationship of features. Therefore, equivariant properties can be achieved by aggregating the features from different positions. On the other hand, an adaptive fusion convolution (AFC) module is designed for the proposed Ms-CapsNet. The higher semantic features can be captured for the primary capsules. Feature extracted by the AFC module significantly improves the robustness to speckle noise. The effectiveness of the proposed Ms-CapsNet is verified on three real SAR data sets. The comparison experiments with four state-of-the-art methods demonstrate the efficiency of the proposed method. Our codes are available at https://github.com/summitgao/SAR_CD_MS_CapsNet.

In recent years, the hyperspectral image (HSI) classification based on generative adversarial networks (GANs) has achieved great progress. GAN-based classification methods can mitigate the limited training sample dilemma to some extent. However, several studies have pointed out that existing GAN-based HSI classification methods are heavily affected by the imbalanced training data problem. The discriminator in GAN always contradicts itself and tries to associate fake labels to the minority-class samples and, thus, impair the classification performance. Another critical issue is the mode collapse in GAN-based methods. The generator is only capable of producing samples within a narrow scope of the data space, which severely hinders the advancement of GAN-based HSI classification methods. In this article, we proposed an Adaptive DropBlock-enhanced Generative Adversarial Networks (ADGANs) for HSI classification. First, to solve the imbalanced training data problem, we adjust the discriminator to be a single classifier, and it will not contradict itself. Second, an adaptive DropBlock (AdapDrop) is proposed as a regularization method employed in the generator and discriminator to alleviate the mode collapse issue. The AdapDrop generated drop masks with adaptive shapes instead of a fixed size region, and it alleviates the limitations of DropBlock in dealing with ground objects with various shapes. Experimental results on three HSI data sets demonstrated that the proposed ADGAN achieved superior performance over state-of-the-art GAN-based methods. Our codes are available at https://github.com/summitgao/HC_ADGAN.

Neural style transfer aims to create artistic images by synthesizing patterns from a given style image. Recently, the Adaptive Instance Normalization (AdaIN) layer is proposed to achieve real-time arbitrary style transfer. However, we observed that if crucial features based on AdaIN can be further emphasized during transfer, both content and style information will be better reflected in stylized images. Furthermore, it is always essential to preserve more details and reduce unexpected artifacts in order to generate appealing results. In this paper, we introduce an improved arbitrary style transfer method based on the self-attention mechanism. A self-attention module is designed to learn what and where to emphasize in the input image. In addition, an extra Laplacian loss is applied to preserve structure details of the content while eliminating artifacts. Experimental results demonstrate that the proposed method outperforms AdaIN and can generate more appealing results.

Convolutional neural networks (CNN) have made great progress for synthetic aperture radar (SAR) images change detection. However, sampling locations of traditional convolutional kernels are fixed and cannot be changed according to the actual structure of the SAR images. Besides, objects may appear with different sizes in natural scenes, which requires the network to have stronger multi-scale representation ability. In this paper, a novel Deformable Residual Convolutional Neural Network (DRNet) is designed for SAR images change detection. First, the proposed DRNet introduces the deformable convolutional sampling locations, and the shape of convolutional kernel can be adaptively adjusted according to the actual structure of ground objects. To create the deformable sampling locations, 2-D offsets are calculated for each pixel according to the spatial information of the input images. Then the sampling location of pixels can adaptively reflect the spatial structure of the input images. Moreover, we proposed a novel pooling module replacing the vanilla pooling to utilize multi-scale information effectively, by constructing hierarchical residual-like connections within one pooling layer, which improve the multi-scale representation ability at a granular level. Experimental results on three real SAR datasets demonstrate the effectiveness of the proposed DR-Net.

High-quality sea ice monitoring is crucial to navigation safety and climate research in the polar regions. In this letter, a transferred multilevel fusion network (MLFN) is proposed for sea ice change detection from synthetic-aperture radar (SAR) images. Considering the fact that training data are limited in the task of sea ice change detection, a large data set was used to train the MLFN, and the deep knowledge can be transferred to sea ice analysis. In addition, cascade dense blocks are employed to optimize the convolutional layers. Multilayer feature fusion is introduced to exploit the complementary information among low-, mid-, and high-level feature representations. Therefore, more discriminative feature extraction can be achieved by the MLFN. Furthermore, the fine-tune strategy is utilized to optimize the network parameters. The experimental results on two real sea ice data sets demonstrated that the proposed method achieved better performance than other competitive methods.

Sea ice change detection from synthetic aperture radar (SAR) images can be regarded as a classification procedure, in which pixels are classified into changed and unchanged classes. However, existing methods usually suffer from the intrinsic speckle noise of multitemporal SAR images. To solve the problem, this letter presents a change detection method based on convolutional-wavelet neural networks (CWNNs). In CWNN, dual-tree complex wavelet transform is introduced into convolutional neural networks for changed and unchanged pixels' classification, and then, the effect of speckle noise is effectively reduced. In addition, a virtual sample generation scheme is employed to create samples for CWNN training, and the problem of limited samples is alleviated. Experimental results on two real SAR image data sets demonstrate the effectiveness and robustness of the proposed method.

Sea ice change detection from synthetic aperture radar (SAR) images is important for navigation safety and natural resource extraction. This paper proposed a sea ice change detection method from SAR images based on collaborative representation. First, neighborhood-based ratio is used to generate a difference image (DI). Then, some reliable samples are selected from the DI by hierarchical fuzzy C-means (FCM) clustering. Finally, based upon these samples, collaborative representation method is utilized to classify pixels from the original SAR images into unchanged and changed class. From there, the final change map can be obtained. Experimental results on two real sea ice datasets demonstrate the superiority of the proposed method over two closely related methods.

Hyperspectral imagery has evident advantages for sea ice classification due to enormous spectral bands. In this paper, we proposed a novel sea ice classification framework from hyperspectral image based on self-paced boost learning (SPBL). First, the criterion of linear prediction error is used for unsupervised band selection. Then, local binary pattern (LBP) features are extracted from the selected bands. Finally, SPBL is employed as the classifier to provide probability outputs using the extracted features. The proposed framework can capture the intrinsic inter-class discriminative models while ensuring the reliability of the samples involved in learning. The experimental results in real-world dataset demonstrate that the proposed framework is superior to several closely related methods.

Hyperspectral image (HSI) classification is a popular yet challenging research topic in the remote sensing community. This letter attempts to encode both spectral and spatial information into deep features for HSI classification. We first propose a semisupervised method for training the stacked autoencoder to obtain discriminative deep features. A batch training scheme is introduced to constrain the label consistency on a neighborhood region. Second, a mean pooling procedure is suggested to further fuse the spectral and local spatial information for deep feature generation. The experimental results on two hyperspectral scenes show that the proposed method achieves promising classification performance.

This letter presents a novel change detection method for multitemporal synthetic aperture radar images based on PCANet. This method exploits representative neighborhood features from each pixel using PCA filters as convolutional filters. Thus, the proposed method is more robust to the speckle noise and can generate change maps with less noise spots. Given two multitemporal images, Gabor wavelets and fuzzy c-means are utilized to select interested pixels that have high probability of being changed or unchanged. Then, new image patches centered at interested pixels are generated and a PCANet model is trained using these patches. Finally, pixels in the multitemporal images are classified by the trained PCANet model. The PCANet classification result and the preclassification result are combined to form the final change map. The experimental results obtained on three real SAR image data sets confirm the effectiveness of the proposed method.

Pansharpening technique is very important for many remote sensing applications. Many fusion algorithms have been proposed to pan-sharpen multispectral (MS) images. However, there are still some spatial or spectral distortion problems in fusion result. There are two major reasons: First, panchromatic (PAN) image contains some interference spectrum information similar to MS image which may cause color distortion in the fusion result. Second, MS image has some interference spatial information approximates PAN image which may cause spatial artifacts. It is difficult to simultaneous eliminate the interference information from PAN and MS images. To solve the above problems, the paper presents an improved pan-sharpen algorithm which integrates the advantages of the ratio enhancement method and Gaussian-fitting. The high-frequency information of each ithband of MS image and the low-frequency information of PAN image are extracted by Gaussian-fitting, and the information is synthesized into a group of low-resolution PAN images. Finally, each ithband of MS image is pan-sharpened by a ratio enhancement, in which the ratio is obtained by image division between the PAN image and the ithsynthesized low-resolution PAN image. Extensive experiments have been implemented on WorldView-2 images. Visual comparison and quantitative analysis demonstrated that the proposed method can achieve good performance in spatial and spectral fidelity.

Automatic ship detection from high-resolution optical satellite images has attracted great interest in the wide applications of maritime security and traffic control. However, most of the popular methods have much difficulty in extracting targets without false alarms due to the variable appearances of ships and complicated background. In this paper, we propose a ship detection approach based on visual search mechanism to solve this problem. First, salient regions are extracted by a global contrast model fast and easily. Second, geometric properties and neighborhood similarity of targets are used for discriminating the ship candidates with ambiguous appearance effectively. Furthermore, we utilize the SVM algorithm to classify each image as including target(s) or not according to the LBP feature of each ship candidate. Extensive experiments validate our proposed scheme outperforms the state-of-the-art methods in terms of detection time and accuracy.

Automatic ship detection in high-resolution optical satellite images with various sea surfaces is a challenging task. In this letter, we propose a novel detection method based on sea surface analysis to solve this problem. The proposed method first analyzes whether the sea surface is homogeneous or not by using two new features. Then, a novel linear function combining pixel and region characteristics is employed to select ship candidates. Finally, Compactness and Length-width ratio are adopted to remove false alarms. Specifically, based on the sea surface analysis, the proposed method cannot only efficiently block out no-candidate regions to reduce computational time, but also automatically assign weights for candidate selection function to optimize the detection performance. Experimental results on real panchromatic satellite images demonstrate the detection accuracy and computational efficiency of the proposed method.

This paper proposes a novel ship detection method based on analyzing the sea state in optical images. This method is composed of three phases. First, the image is segmented with the improved region splitting and merging method, which divides the sea into separated regions. Then, the sea state of each divided region of sea is analyzed by extracting texture roughness and ripple density of a modified differential box counting (DBC) method. Finally, an appropriate algorithm is applied to detect ships for each region of sea. Experimental results test on 36 real remote sensing images and 133 images obtained from Google earth demonstrate that the method is free of image resolution and has little limitation of sea conditions.

Nowadays, with the rapid increase of spatial resolution in remote sensing, accurate and efficient identification of man-made targets from large-scale high resolution panchromatic images plays a more and more important role in real applications. This paper presents an efficient bridge extraction method based on CDT. it consists of two steps: river segmentation and bridge extraction. First, river region is segmented from complicated background by texture analysis. Second, constrained Delaunay triangulation (CDT) is applied to segment river region into a triangular mesh. The medial axes of river is computed from the skeleton segments of triangles and therefore bridges are easily detected along the end points of river medial axes. Extensive experiments have been performed on panchromatic QuickBird images. The experiment results show that our method achieves detection accuracy and satisfies the speed request in large-scale high resolution panchromatic image.

Buildings, roads, and squares are all man-made objects which have similar spectral feature. The similarity leads high false alarm rate of building extraction methods that only depend on spectral feature. In high resolution multispectral images, the shadow of buildings are visible, thus, based on the relations of buildings and shadow, this paper proposes a shadow detection method, extracts building sample points exactly depending on shadow using the similarity of angle between shade line and shadow line in one image, then perfectly achieves building extraction. The method reduces the difficulty of classifying buildings from other similar man-made objects, moreover, the detection accuracy is greatly improved.