Multimodal image registration is a key procedure in remote sensing applications (such as remote sensing image stitching), which faces significant challenges including radiometric discrepancies and local geometric deformations caused by the differences of both sensor and imaging parameters. Traditional methods remove coarse error using global features, making it difficult to identify misregistrations at early stage, thus limiting registration accuracy improvement. When existing convolutional registration neural networks extract deep features, shallow local feature information is usually lost because the network gradually focuses on high-level abstract features, causing local details to be simplified or lost in the global feature construction. Solving this problem will greatly increase the complexity of the model, and the network needs to reorganize and train the data according to specific tasks, which is time-consuming. To address these issues, this letter develops a hybrid registration model with a global-local descriptor. Specifically, we first obtain improved RIFT keypoints via combining rotated and scale invariant corner points produced by the integral scale detection Min-moment with extracted edge points generated by the FAST detection Max-moment. Then, a global-local descriptor is constructed by combining the improved RIFT descriptor with the LoFTR coarse-grained feature descriptor. Finally, a 0–1 distance allocation matrix is formulated to improve the registration success rate (SR). The experimental results show that the proposed method has a powerful capability in improving both generalization and accuracy and outperforms mainstream methods, even the average number of correctly registered correspondences is about two times and 1.7 times higher than LoFTR and RIFT, respectively.

The widespread use of time delay and integration charge-coupled device (TDI CCD) technology in high-resolution spaceborne optical cameras has made high-frequency jitter effects a common issue, resulting in different levels of distortion in images. Current methods mostly concentrate on correction of obviously high levels of geometric distortion. Focusing on low levels of geometric distortion, which are more difficult to accurately detect, this paper proposes a progressive learning-based correction method for high-frequency jitter distortion in remote sensing images from spaceborne TDI CCD cameras, utilizing a Generative Adversarial Network (GAN). First, a distorted dataset with diverse jitter levels for progressive training is generated through jitter simulation model by adjusting the parameters. Then, a GAN model is employed for the correction task. The generator consists of the Distortion Net for geometric distortion correction and the Detail Enhancement Net for image detail restoration. Finally, a progressive learning strategy is used to gradually enhance the ability of network to correct minor geometric distortion. The proposed method is validated using simulated images and real-world satellite images. Experimental results demonstrate that the proposed method outperforms existing restoration methods both in simulated datasets and practical scenarios.

This paper presents a Generative Adversarial Network (GAN)-based jitter distortion correction method for spaceborne images of Time Delay Integration (TDI) Charge-Coupled Device (CCD) camera. This method leverages the advantages of GANs and combines content loss, adversarial loss, and perceptual loss to effectively repair distorted images while preserving image details, which does not rely on jitter information captured by high-frequency attitude sensors, nor depends on the analysis of overlapping areas between different bands in multispectral images. The experimental results show that the proposed method achieves automated correction of geometric distortions and has shown promising restoration results on real distorted images captured by Yaogan-26 satellite and GaoFen satellite, which achieves better results than other blind restoration methods.

Infrared images are susceptible to thermal radiation. Infrared thermal radiation correction methods based on physical prior may fail while correcting real-world images, because assumed priors do not always hold in the real world, resulting in the presence of thermal radiation residuals. Supervised learning-based methods have the potential to achieve favorable outcomes in the correction of synthetic images. However, due to the unavailability of labeled datasets, their efficacy is limited when applied to real-world images. To address this problem, in this article, to the best of our knowledge, we propose the first semi-supervised learning network for infrared radiation correction in the real world, named SIRCNet. The network is trained using a semi-supervised strategy, which includes a supervised training stage and a self-supervised training stage. In the supervised training stage, we constructed a multilevel wavelet decomposition and reconstruction correction (MWDRC) module for latent image correction and an efficient generalized feature extraction (EGFE) module for bias field estimation. Furthermore, EGFE is composed of one partial channel interactive (PCI) attention block and three effective residual blocks (ERBs). Surface fitting can approximate the thermal radiation bias field of the thermal radiation degradation images. The fit bias field can provide critical prior knowledge that enhances EGFE’s estimation of the thermal radiation bias field. Hence, in the self-supervised training stage, when fine-tuning MWDRC and EGFE using a generator, surface fitting is employed to constrain EGFE. Comparative experiments demonstrate that SIRCNet outperforms existing correction methods on both real and synthetic datasets, achieving the best metrics as well as visualization.

Recently, infrared thermal radiation effect correction methods are dominated by removing bias field in spatial domain. Since they do not consider the low-frequency characteristics of thermal radiation bias field and the high-frequency information of image content, these methods often fail in the enhancement of contrast and details. To address this problem, we propose a novel wavelet decomposition two-stage network for infrared thermal radiation effect correction, named WDTSNet. Through wavelet decomposition, we construct a low-frequency thermal radiation effect coarse correction subnetwork (LFCCSN) and a high-frequency detail enhancement fine correction subnetwork (HFFCSN), respectively. Firstly, we take the small size low-frequency component of the degraded image after discrete wavelet transformation (DWT) as the input of the first stage LFCCSN and propose an intra-block multiscale residual dense module (IMRDM) to complete the coarse correction and contrast enhancement through different scales of receptive fields and intra-block channel information interaction. Secondly, we perform inverse discrete wavelet transformation (IDWT) to obtain the input of the second stage HFFCSN, and build a high-frequency gated residual module (HGRM) in HFFCSN to remove residual thermal radiation bias field and acquire the enhanced high-frequency information. In addition, we further design dual-branch cross-scale attention fusion module (DCAFM) between encoders and decoders to effectively aggregate the cross-scale information flow. Extensive experiments on simulated and real infrared images demonstrate that the proposed WDTSNet performs well on enhancing contrast and details than existing methods. The code will be publicly available upon acceptance.

Time delay integration charge-coupled device (TDI CCD) is sensitive to the platform’s stability during push-broom imaging. Due to variations in total integration time, panchromatic and multispectral images suffer varying degrees of geometric distortion caused by satellite jitter with high frequency, which leads to different inner distortion in different band images and different band-to-band mismatching errors between different band combinations. To address this problem, this paper proposes a rigorous parallax observation model considering multi-stage integration time and presents a jitter distortion correction method for remote sensing panchromatic and multispectral images captured by TDI cameras based on it. First, the law of the amplitude attenuation and phase offset of platform jitter deviation on the image under different TDI stages is determined through simulation verification. Then, the rigorous parallax observation model is proposed to establish an accurate relationship between the relative jitter error of two multispectral images with multi-stage integration and the absolute single-stage integration jitter error by introducing the amplitude attenuation factor and phase offset. Finally, the jitter distortion curves of images with different integration stages and integration time can be reconstructed based on the estimated absolute jitter error and the imaging parameters. Subsequently, the jitter distortion can be further corrected by image resampling. The proposed method was verified through both simulation and real data experiments using GaoFen-9 satellite images. Experimental results show that the proposed method can effectively correct high-frequency jitter distortion in panchromatic and multispectral images, which cannot be corrected by traditional single-stage integration jitter detection model.

The aim of weakly supervised object co-localization is to locate different objects of the same superclass in a dataset. Recent methods achieve impressive co-localization performance by multiple instance learning and self-supervised learning. However, these methods ignore the common part information shared by fine-grained objects and the influence of the complementary parts on the co-localization of the fine-grained objects. To solve these issues, we propose a complementary parts contrastive learning method for fine-grained weakly supervised object co-localization. The proposed method follows such an assumption that fine-grained object parts with the same/different semantic meaning should have similar/dissimilar feature representations in the feature space. The proposed method tackles two critical issues in this task:$i)$how to spread the model’s attention and suppress the complex background noise, and$ii)$how to leverage the cross-category common parts information to mitigate the context co-occurrence problem. To address$i)$, we attempt to integrate local and context cues via three types of attention including self-supervised attention, channel, and spatial attention to spread the model’s attention toward automatically identifying and localizing most discriminative parts of objects in the fine-grained images. To solve$ii)$, we propose a cross-category object complementarity part contrastive learning module to identify the extracted part regions with different semantic information by pulling the same part features closer and pushing different part features away, which can mitigate the confounding bias caused by the co-occurrence surroundings within specific classes. Extensive qualitative and quantitative evaluations demonstrate the effectiveness of the proposed method on four fine-grained co-localization datasets: CUB-200–2011, Stanford Cars, FGVC-Aircraft, and Stanford Dogs. Code and models are available athttps://github.com/Zhao-fan/CPCL.

Due to the unique imaging mechanism and wide application of synthetic aperture radar (SAR), SAR image interpretation has been researched by more and more scholars. The supervised SAR image semantic segmentation methods that based on deep learning require a large number of accurate pixel-level labels, which are very hard to obtain. The lack of labeled samples limits the practical application of deep learning methods in SAR image semantic segmentation. To reduce the requirement of labeled data, we decided to introduce the cross pseudo-supervision network (CPS-Net) into SAR image semi-supervised semantic segmentation and promote the development of semi-supervised learning in SAR image interpretation. The semi-supervised segmentation based on CPS-Net has the following advantages: (1) CPS-Net encourages high similarity between two networks with the same input data, which helps improve the performance. (2) CPS-Net can make better use of the pseudo-supervision of unlabeled data to guide the network training. Experimental results show that CPS-Net achieves excellent semi-supervised semantic segmentation results on Sentinel-1 dual-polarization data with less labeled data. Compared with well-known semantic segmentation methods U-Net and DeeplabV3+, the performance of SAR image segmentation is significantly improved.

Multi-sensor image registration is a basic and important issue in the field of remote sensing applications. At present, many algorithms have not directly evaluated and analyzed the feature descriptor design of the algorithm. Taking the feature descriptors of RIFT, SIFT, SAR-SIFT and HAPCG as the analysis objects, this paper designs experiments to analyze their stability under gray distortion and local geometric distortion, gives a quantitative evaluation, and reveals the contribution of the feature descriptor of each multi-sensor image registration algorithm in the process of multi-sensor image registration.

Limited labels and increasing multisource data promote domain adaptation (DA) problem as a challenging study for polarimetric synthetic aperture radar (PolSAR) interpretation. Existing DAs for optical images cannot generalize over PolSAR imagery due to its special side-imaging characteristics and complex distribution shifts. In this letter, a general feature paradigm (GFP) is proposed for unsupervised cross-domain PolSAR image classification. The GFP is based on a key observation that interclass aggregation is optimized after four-step feature transformations. This key observation leads to GFP that not only reduces the domain shifts but also compatible with typical DA methods. The GFPs are conducted on both source and target domain by unsupervised manner, including polarimetric basis extraction, the Wishart clustering, histogram statistics, and dimensionality reduction. After these transformations, the unlabeled target PolSAR image can be classified based on obtained GFP, DA, and limited labeled samples only from the source domain. Extensive unsupervised cross-domain experiments on 27 scenarios verified that GFP leads to at most 93.76% accuracy for full- and dual-polarized synthetic aperture radar (SAR) images’ classification. Moreover, the GFP shed light on extensive cross-domain PolSAR applications about built-up areas, vegetation, and bare land analysis.

Polarimetric synthetic aperture radar (PolSAR) image classification is one of the fundamental research areas in remote sensing. Superpixels can provide boundary constraint information and are widely used in PolSAR image interpretation. However, traditional machine learning superpixel algorithms have many limitations for PolSAR image interpretation. Pseudo-color images are usually used as the superpixel algorithm inputs, and the loss of polarimetric information will decrease the performance. In addition, the superpixel algorithms are difficult to incorporate into state-of-the-art deep learning models and cannot be trained in an end-to-end manner. In this letter, a trainable end-to-end deep superpixel network is proposed for PolSAR image classification. The inputs of the proposed method can be any low/middle-level polarimetric features of a PolSAR image and the rich polarimetric feature representation can be learned. The produced superpixels of the proposed method are more concentrated near the land cover boundaries and can significantly improve the performance of PolSAR image classification. Experimental results show that the overall accuracies of the proposed method are approximately 2.57% and 1.44% higher than traditional superpixel algorithms on two PolSAR datasets and surpass some well-known deep learning methods.

Increasing amounts of polarimetric synthetic aperture radar (PolSAR) images from different sensors covering different scenes are available, but limited labeled samples and trained models can hardly work well in these cross-domain data interpretations. Fortunately, domain adaptation (DA) can transfer knowledge in existing images to new yet related images. DA shows attractive potential for PolSAR classification, and it is still challenging due to more complex domain shifts caused by different sensors, imaging conditions, and distributions. Inspired by the widely applicable polarimetric scattering mechanisms and DA ability of subspace alignment (SA), this article is devoted to constructing a robust unsupervised cross-domain PolSAR classification framework, by exploring scattering and statistical characteristics mapping between the source and target domains. First, classical scattering components of both source and target data were extracted, and Wishart clustering was adopted to derive the statistical information of scattering components at patch level. Second, the intrinsic polarimetric scattering components were estimated and extracted, which were called statistical scattering components (SSCs). Third, by applying SA, the source SSC was aligned with target SSC, and domain shift was further reduced. Finally, the target PolSAR image was classified based on labeled samples from source domain, and unsupervised cross-domain classification was achieved by SSC-based SA (SSC-SA). The unsupervised cross-domain experiments are conducted on 49 units among 11 data sets, including Radarsat-2, Gaofen-3, AIRSAR, and Pi-SAR images. With randomly selected labeled samples (about 2%–10%) from source domain, the accuracies of the proposed cross-domain classifications range between 80.20% and 95.64%. Also, the proposed SSC feature pattern is proved extensible for other polarimetric basis and decompositions.

Supervised deep learning models usually need large amounts of labeled data due to the data-driven training strategies, and its applicability to the newly emerging categories that lack annotated images is severely limited. In contrast, few-shot learning aims to recognize novel targets from very few labeled examples, so it will be a promising method for synthetic aperture radar (SAR) image interpretation, where numerous labeled data may not exist. In this paper, we introduced a few-shot learning method based on relation network and graph neural network (GNN). Relation network extracts the feature similarity between query samples and support samples through a convolutional neural network, and it has achieved good performance in few-shot learning problems. GNNs have received increasing attention in recent years, and they have shown superior performance in relation extraction. In this work, we replaced the relation module in the relation network with attention GNN, aiming to model the relationship between the samples more effectively and learn a better metric for feature similarity. Experiments on the MSTAR dataset demonstrate that the proposed method can better extract the relationship between query samples and support samples, thereby improving the performance for few-shot image classification tasks.

Accurate built-up area (BA) information plays crucial role for many applications. PolSAR imagery can provide important source for BAs information analysis. However, the BAs with large orientation angles are usually misdetected as vegetation, and labeled BA samples with special orientations are diffi-cult to obtain. In this paper, a PolSAR BA extraction method based on eigenvalue statistical information and PU-Learning is proposed to overcome abovementioned problems. Firstly, the roll invariance of coherency-matrix eigenvalues and the building orientations have been analyzed. Then, by adopting eigenvalue-Wishart unsupervised classification, regional statistical information and rotation invariant property are comprehensively utilized. Finally, the BAs are extracted by combining PU-Learning classifier with only positive samples at same distinguishable orientation. Six experiments on PolSAR imageries show the accuracy of proposed method can reach 92-99% with only a few positive samples, 8-20% higher than classical model decomposition-based PU-Learning method, and the requirement for labeled samples is less than 0.65%.

Polarimetric synthetic aperture radar (PolSAR) image classification is one of the most important applications in Pol-SAR image processing. More and more deep learning methods are applied to PolSAR image classification. As we know, the polarimetric response of a target is related to the orientation of the target, but the features in rotation domain are not fully used in deep learning. We use a convolutional LSTM (ConvLSTM) along with a sequence of polarization coherent matrices in rotation domain for PolSAR image classification. First, nine different polarization orientation angles (POA) are used to generate nine polarization coherent matrices in rotation domain. Second, a deep learning model that stacked with multiple ConvLSTM layers and fully connected layers is proposed for classification. Finally, the sequence of polarization coherent matrices is fed into the ConvLSTM to classify Pol-SAR images. Experiments show that the classification results of ConvLSTM are better than the LeNet-5.