Synthetic aperture radar (SAR)-optical data fusion cloud removal is a highly promising cloud removal technology that has attracted considerable attention. In this field, primary researches are based on deep learning, which can be divided into two categories: convolutional neural network (CNN)-based and Transformer-based. In the cases of extensive cloud coverage, CNN-based methods, with their local spatial awareness, effectively capture local structural information in SAR data, preserving clear contours of cloud-removed land covers. However, these methods struggle to capture global land cover information in optical images, often resulting in notable color discrepancies between recovered and cloud-free regions. Conversely, Transformer-based methods, with their global modeling capability and inherent low-pass filtering properties, excel at capturing long-range spatial correlations in optical images, thereby maintaining color consistency across cloud-removal outputs. However, these methods are less effective at capturing the fine structural details in SAR data, which can lead to blurred local contours in the final cloud-removed images. In this context, a novel framework called heterogeneous parallel network for cloud removal (HPN-CR) is proposed to achieve high-quality cloud removal under extensive cloud coverage conditions. HPN-CR employs the proposed heterogeneous encoder with its SAR-optical input approach to effectively extract and fuse the local structural information in cloudy areas from SAR images, with the spectral information about land covers in cloud-free areas from the whole optical images. In particular, it uses a ResNet network with local spatial awareness to extract SAR features. It also uses the proposed Decloudformer, which globally models multiscale spatial correlations, to extract optical features. The output features are fused by the heterogeneous encoder and then reconstructed to cloud-removal images through a pixelshuffle-based decoder. Comprehensive experiments were conducted and the experimental results demonstrated the effectiveness and superiority of the proposed method. The code is available athttps://github.com/G-pz/HPN-CR.

At present, most remote sensing image change detection methods are applicable to equal-resolution scenarios, that is, the bi-temporal images are assumed to have the same spatial resolution. Real-world tasks such as disaster emergency response have put forward the need for change detection of bi-temporal images with different spatial resolutions, that is, cross-resolution change detection. However, due to the significant differences in spatial details between high-resolution images and low-resolution images, it is difficult to extract the spatial features of changed landcovers in complex scenarios and distinguish between changed landcovers and unchanged landcovers. To overcome the above issues, we propose a Resolution-Difference Embedding Network (RDENet). RDENet combines two innovative methods: pseudo-continuous resolution sequence representation (PRSR) and bi-temporal resolution-difference modulation (BRDM). The PRSR method effectively extracts features of landcovers in complex scenarios by constructing pseudo image sequence with smooth transition of resolution and developing a resolution-guided feature fusion module. The BRDM method significantly enhances the model’s ability to distinguish between changed and unchanged landcovers in complex scenarios through the design of a resolution-aware self-modulation module and resolution-aware mutual-modulation module, which utilizes the resolution-difference factor as prior knowledge to dynamically enhance key features of changed landcovers in cross-resolution image pairs. Extensive experiments conducted on three publicly available change detection datasets demonstrate that the proposed RDENet achieves superior detection performance in cross-resolution scenarios.

Multimodal remote sensing (RS) image land use/land cover (LULC) classification using optical and synthetic aperture radar (SAR) images has raised attention for recent studies. Current methods primarily employ multimodal fusion operations to directly explore relationships between multimodal features and obtain fused features, leading to the loss of beneficial modality-specific information problem. To solve this problem, this study introduces a multimodal feature decomposition and fusion (MDF) approach combined with a visual state space (VSS) block, namely MDF-VSS block. The MDF-VSS block emphasizes beneficial modality-specific information and perceives shared land cover information through modality-difference and modality-share features, which are then adaptively integrated to obtain discriminative fused features. Based on the MDF-VSS block, an MDF decoder is designed to retain beneficial multi-scale modality-specific information. Then, a multimodal specific information enhancement (MSIE) decoder is designed to perform modality-difference feature guided auxiliary classification tasks, further enhancing modality-specific information that is expert in classification. Combining the MDF and MSIE decoders, a novel retain-enhance fusion network (REF-Net) is proposed to retain and enhance modality-specific information that benefits classification, thus improving the performance of multimodal RS image LULC classification. Extensive experimental results obtained on three public datasets demonstrate the effectiveness of the proposed REF-Net. The source code will be available at https://github.com/TINYWAI/REF_Net.

At present, deep learning (DL) methods for remote sensing (RS) change detection (CD) are developing rapidly. However, there are still many challenges in complex RS scenarios. Factors, such as season and illumination, contribute to minimal differences in radiation characteristics between the changed area and the background, making them difficult to distinguish. This letter proposes an axial-shift feature interaction and prototype-guided penalty constraint network (ASPGNet) to address this problem. ASPGNet integrates axial-shift feature interaction (ASFI) module and prototype-guided penalty constraint (PGPC) loss. The ASFI module facilitates interaction among adjacent features through axial-shift operations in the width/height directions, aiming to obtain discriminative feature representations of the changed area. The PGPC loss utilizes prototypes to adaptively identify and weigh confusing pixel features, ensuring distinguishability between change and nonchange features and, thereby, generating accurate CD results. We evaluate the proposed method on the WHU-CD and LEVIR-CD datasets, achieving the$F1$scores of 93.22% and 91.49%, respectively. These results demonstrate the effectiveness of the proposed method.

Scene classification of optical remote sensing is a basic but important task because of its broad application in a range of fields. Due to the powerful feature extraction capabilities, convolutional neural networks (CNNs) have been widely used in optical remote sensing scene classification tasks. Despite the remarkable efforts have been achieved, there are still several problems existed, including the effective multiscale feature description for complex scene, rotation, and low interclass diversity problems. In this letter, to address these mentioned problems and construct a powerful CNN for optical remote sensing scene classification, an effective multiscale residual network with a high-order feature representation (MRHNet) is proposed. First, data preprocessing is utilized to adapt the rotation invariance problem. Second, related to the original residual module, a pyramid convolution is introduced to realize the multiscale feature extraction, and then, its feature description ability is further improved by an effective channel attention module. Third, inspired by the tensor decomposition and its completion, a high-order feature representation structure is designed for recovering discriminative fine-scale details into deep layers to solve the low interclass diversity problem. Finally, extensive experiments are carried on two widely used scene classification datasets (e.g., AID and NWPU-RESISC45), and comparing results show that the proposed MRHNet can achieve superior performances.

Scene classification has become an active research area in remote sensing (RS) image interpretation. Recently, Transformer-based methods have shown great potential in modeling global semantic information and have been exploited in RS scene classification. In this letter, we propose a multi-level fusion Swin Transformer (MFST), which integrates a multi-level feature merging (MFM) module and an adaptive feature compression (AFC) module to further boost the performance for RS scene classification. The MFM module narrows the semantic gaps in multi-level features via patch merging in lower-level feature maps and lateral connections in the top-down pathway. The AFC module makes multi-level features have smaller dimensions and more coherent semantic information by adaptive channel reduction. We evaluate the proposed network on the aerial image dataset (AID) and NWPU-RESISC45 (NWPU) datasets, and the classification results reveal that the proposed network outperforms several state-of-the-art (SOTA) methods.

Recently, convolutional neural network (CNN)-based remote sensing scene classification has achieved great success. However, the prohibitively expensive computation and storage requirements of state-of-the-art models have hindered the deployment of CNNs on on- board platforms. In this letter, we propose a differentiable neural architecture search (NAS)-based channel pruning method to automatically prune the CNN models. In the proposed method, the importance of each output channel is measured by a trainable score. The scores are optimized by an NAS method to search a good-performance pruned structure. After the search process, a global score threshold is adopted to derive the pruned model. A cost-awareness loss is proposed for the search process to encourage the floating-point operation (FLOP) compression ratio of the pruned model coverage to a desired value. We apply the proposed method to ResNet-34 and VGG-16 to verify the performance. The NWPU-RESISC-45 and UC Merced Land-Use (UCM) datasets are used for the performance evaluation. A comparison with state-of-the-art pruning methods demonstrates that the proposed method can achieve competitive performance with a similar reduction in FLOP.

In practical applications, remote-sensing scene classification tasks generally exhibit data shift problems. In this situation, images have large data discrepancies, leading to diffused features and performance degradation. To address the data shift problem, we propose cosine margin prototypical networks. Specifically, we adopt a cosine margin to constrain strictly features, generating well-clustered and discriminative features. With the cosine margin, our method can alleviate data discrepancies by obtaining discriminative features and further addresses the data shift problem well. We conduct extensive experiments on various datasets and achieve 0.03%–7.11% higher accuracy than existing methods. Competitive experimental results demonstrate that our method can solve the data shift problems well in remote-sensing scene classification.

Currently, remote-sensing (RS) scene classification has played an important role in many practical applications. However, traditional methods that are based on deep convolutional neural networks (DCNNs) have several difficulties when faced with data shift problems: novel classes, varied orientations, and large intraclass variations of RS scene images. In this letter, we propose the rotation-invariant and discriminative-learning prototypical networks (RDPNs) for RS scene classification. RDPN uses A-ORConv32 basic blocks and attention mechanisms to obtain rotation-invariant and discriminative features. In addition, adaptive cosine center loss is proposed to constrain the features to mitigate the large intraclass variations and penalize the hard samples adaptively. We conduct extensive experiments on publicly available datasets and achieve 1.69%–19.38% higher accuracy than existing methods. The experimental results verify that the proposed RDPN can solve the data shift problems well in RS scene classification.

The power of features considerably influences the classification performance of remote sensing scene classification (RSSC). Recently, deep convolutional neural networks (DCNNs) have been used to extract powerful scene features. Nevertheless, confusion and overlap still occur in the feature space, leading to inaccurate RSSC. To alleviate this problem, we propose a novel deep metric learning loss function incorporated into a sphere loss to enhance the discrimination of feature representations. Inspired by two representative loss functions (i.e., angular loss and center loss), the proposed sphere loss learns a unique cluster center for each class in a remote sensing scene. Because the cluster centers and features are restricted by an introduced geometrical constraint, the intraclass distance of features decreases, while the interclass distance increases. Moreover, we introduce a spatial constraint, i.e., a uniformity coefficient on different cluster centers, which causes the centers to form a uniform distribution that maximizes the interclass distances between features. Extensive analysis and experiments on three commonly used RSSC data sets consistently show that, compared with state-of-the-art methods, the proposed sphere loss can effectively learn discriminative feature representations and significantly improve RSSC.

Extensive convolutional neural network (CNN)-based methods have been widely used in remote sensing scene classification. However, the dense operation and huge memory storage of the state-of-the-art models hinder their deployment on low-power embedded devices. In this letter, we propose a mixed-precision quantization method to compress the model size without accuracy degradation. In this method, we propose a symmetric nonlinear quantization scheme to reduce the quantization error. A corresponding three-step training strategy is proposed to improve the performance of the quantized network. Finally, based on the proposed scheme and training strategy, we propose a neural architecture search (NAS)-based quantization bit-width search (NQBS) method. This method can automatically select a bit width for each quantized layer to obtain a mixed-precision network with an optimal model size. We apply the proposed method to the ResNet-34 and SqueezeNet networks and evaluate the quantized networks on the NWPU-RESISC45 data set. The experimental results show that the mixed-precision quantized networks under the proposed method strike a satisfying tradeoff between classification accuracy and model size.

Recently, remote sensing image scene classification technology has been widely applied in many applicable industries. As a result, several remote sensing image scene classification frameworks have been proposed; in particular, those based on deep convolutional neural networks have received considerable attention. However, most of these methods have performance limitations when analyzing images with large intraclass variations. To overcome this limitation, this letter presents the marginal center loss with an adaptive margin. The marginal center loss separates hard samples and enhances the contributions of hard samples to minimize the variations in features of the same class. Experimental results on public remote sensing image scene data sets demonstrate the effectiveness of our method. After the model is trained using the marginal center loss, the variations in the features of the same class are reduced. Furthermore, a comparison with state-of-the-art methods proves that our model has competitive performance in the field of remote sensing image scene classification.

Automatical inshore ship change detection in optical remote sensing images is meaningful for a wide range of applications, but still a challenging task. In this paper, a visual search inspired model is proposed for inshore ship change detection. The proposed model achieves inshore ship change information extraction based on spatial-temporal saliency detection which mimics visual search mechanism. Experimental results performed on a Google Earth optical remote sensing image data set demonstrate the effectiveness of the proposed model in terms of visual and objective evaluations.

For water resource analysis applications, due to very high resolution and large observation scope, optical remote sensing images can present more visible object characters. Water-land segmentation from optical remote sensing images is wildly used and becomes a hot research topic. However, since large scale complex background scenes include many interferences, the water-land segmentation from optical remote sensing images becomes a challenge task. Aim to achieve better water area feature description from complex land cover background, we apply a sub-neighbor system convolutional neural network (SNS-CNN) to the water-land segmentation in harbor scene areas. First, on the basis of the U-net structure, an optimized up-sampling process is proposed to enhance water area feature expression. Second, a novel sub-neighbor system constraint of each predicted pixel point is leaded into the loss function to make the model producing water mask more coherent. Furthermore, experiments on our collected variety of optical remote sensing images demonstrate that this paper proposed water-land segmentation method can produce better performance than state of the art methods.

Object detection in complex optical remote sensing images is a challenging problem due to the wide variety of scales, densities, and shapes of object instances on the earth surface. In this letter, we focus on the wide-scale variation problem of multiclass object detection and propose an effective object detection framework in remote sensing images based on YOLOv2. To make the model adaptable to multiscale object detection, we design a network that concatenates feature maps from layers of different depths and adopt a feature introducing strategy based on oriented response dilated convolution. Through this strategy, the performance for small-scale object detection is improved without losing the performance for large-scale object detection. Compared to YOLOv2, the performance of the proposed framework tested in the DOTA (a large-scale data set for object detection in aerial images) data set improves by 4.4% mean average precision without adding extra parameters. The proposed framework achieves real-time detection for$1024\times 1024$image using Titan Xp GPU acceleration.11https://github.com/WenchaoliuMUC/Detection-of-Multiclass-Objects-in-Optical-Remote-Sensing-Images

In this paper, an efficient and accurate multilevel change detection method is presented, achieved by the combination of an unsupervised object-based correlation analysis and a supervised post-classification comparison. Before the change detection procedure, fast multitemporal segmentation is applied to provide object-level information on two registered images. Then the proposed object-based correlation analysis method is used to extract the potential changed areas efficiently and stably, which improves the accuracy of overall performance. Notably, all the procedures are highly automatic except for the necessary selection of training examples within all supervised algorithms. The experimental results demonstrate the superior performance of our method compared with the four typical state-of-the-art change detection methods.

Ship detection is a challenging problem in complex optical remote-sensing images. In this letter, an effective ship detection framework in remote-sensing images based on the convolutional neural network is proposed. The framework is designed to predict bounding box of ship with orientation angle information. Note that the angle information which is added to bounding box regression makes bounding box accurately fit into the ship region. In order to make the model adaptable to the detection of multiscale ship targets, especially small-sized ships, we design the network with feature maps from the layers of different depths. The whole detection pipeline is a single network and achieves real-time detection for a $704 \times 704$ image with the use of Titan X GPU acceleration. Through experiments, we validate the effectiveness, robustness, and accuracy of the proposed ship detection framework in complex remote-sensing scenes.

In recent times, many efforts have been made to improve remote sensing image scene classification, especially using popular deep convolutional neural networks. However, most of these methods do not consider the specific scene orientation of the remote sensing images. In this letter, we propose the improved oriented response network (IORN), which is based on the ORN, to handle the orientation problem in remote sensing image scene classification. We propose average active rotating filters (A-ARFs) in the IORN. While IORNs are being trained, A-ARFs are updated by a method that is different from the ARFs of the ORN, without additional computations. This change helps IORN improve its ability to encode orientation information and speeds up optimization during training. We also propose Squeeze-ORAlign (S-ORAlign) by adding a squeeze layer to ORAlign of ORN. With the squeeze layer, S-ORAlign can address large-scale images, unlike ORAlign. An ablation study and comparison experiments are designed on a public remote sensing image scene classification data set. The experimental results demonstrate the effectiveness and better performance of the proposed model over that of other state-of-the-art models.

Sea-land segmentation is a key step for some important applications of panchromatic remote sensing image processing. However, robust and effective sea-land segmentation for high-resolution panchromatic remote sensing images is still a challenging problem. This letter presents an accurate and robust approach by integrating the improved multiscale normalized cut (IMNcut) method and improved Chan-Vese model for sea-land segmentation. At first, the image is downsampled and segmented into multiple regions by the IMNcut method. Next, the homogeneous regions are merged to obtain a coarse segmentation result. Finally, gray intensity and local entropy features are integrated as discriminants of the improved Chan-Vese model, which is used to obtain the final segmentation result through a low- to high-resolution segmentation scheme. Experimental results performed on several real data sets demonstrate the effectiveness of the proposed model in terms of visual and objective evaluations.