In the non-cooperative electromagnetic environment, new radar emitters will emerge unexpectedly during the test phase, which brings the “Open-Set” Radar Emitter Recognition (OS-RER). Conventional classifiers cannot identify new radar emitters that do not exist in the training dataset. Therefore, in this paper, a novel Deep Metric Auto-Encoder (DMAE) is proposed for OS-RER. In DMAE, deep metric learning learns new non-linear mappings in the metric space to measure the similarity between instances. The dual-path deep auto-encoder is designed to reduce the open space risk by learning a low-dimensional manifold and a discriminative representation of known instances. Specifically, DMAE models known classes, and measures class belongingness through the reconstruction error of the AE and the entropy of the classifier. The deep metric network learns a more precise distance metric by minimizing the distance between the known class instances and the corresponding reconstruction. To accurately detect unknown instances, the classifier and the deep metric network are used together to preliminarily detect unknown instances. Finally, the detected unknown instances are used to further train the classifier to recognize the radar emitter in the open-set scenarios. The DMAE learns the discriminative representation through end-to-end learning. Extensive experiments conducted on real radar datasets and simulated radar datasets show that DMAE can identify unknown emitters and significantly outperforms existing open-set classification methods.

The integration of spatial and spectral information is beneficial to the improvement of change detection (CD) performance. However, existing methods cannot efficiently suppress the influences of spatial and spectral differences (SDs) in unchanged areas. To address these issues, in this article, we propose a content-guided spatial–spectral integration network (CSI-Net) for the fusion of global spatial details and SD information. Specifically, the proposed CSI-Net is composed of a spatial reasoning (SR) module, an SD module, and a content-guided integration (CGI) module. In the SR module, the spatial information is learned by cascaded graph convolution (GC) blocks for global modeling. The SD module is responsible for the extraction of spectral features, by calculating the means and variances of features to reduce the impact of SDs in unchanged regions. In addition, in order to integrate the spatial–spectral features efficiently, we design a CGI module to further take advantage of their complementary information. In this module, high-level content information is introduced as a guide for proper interaction. Due to the efficient spatial–spectral fusion, the proposed CSI-Net can learn the changed features better while achieving suppression of SDs. Experimental results on LEVIR-CD, WHU-CD, and CLCD datasets demonstrate that the proposed CSI-Net produces better performance compared to state-of-the-art methods, and is applicable to different scenarios. The code of CSI-Net is available athttps://github.com/RSMagneto/CSI-Net.

Recently, deep neural network (DNN)-based methods have achieved good results in terms of the fusion of low spatial resolution hyperspectral (LR HS) and high spatial resolution multispectral (HR MS) images. However, the spectral band correlation (SBC) and the spatial nonlocal similarity (SNS) in hyperspectral (HS) images are not sufficiently exploited by them. To model the two priors efficiently, we propose a spectral-spatial dual graph unfolding network (SDGU-Net), which is derived from the optimization of graph regularized restoration models. Specifically, we introduce spectral and spatial graphs to regularize the reconstruction of the desired high spatial resolution hyperspectral (HR HS) image. To explore the SBC and SNS priors of HS images in feature space and utilize the powerful learning ability of DNNs simultaneously, the iterative optimization of the spectral and spatial graph regularized models is unfolded as a network, which is composed of spectral and spatial graph unfolding modules. The two kinds of modules are designed according to the solutions of the spectral and spatial graph regularized models. In these modules, we employ graph convolution networks (GCNs) to capture the SBC and SNS in the fused image. Then, the learned features are integrated by the corresponding feature fusion modules and fed into the feature condense module to generate the HR HS image. We conduct extensive experiments on three benchmark datasets and the results demonstrate the effectiveness of our proposed SDGU-Net.

In this article, we propose a new pan-sharpening method that disentangles low spatial resolution multispectral (LRMS) and panchromatic (PAN) images in terms of sensor-specific features and common features. These features are obtained by defining mutual information (MI)-based transformers designed to achieve disentangled learning. In the proposed method, LRMS and PAN images are cross-reconstructed by cross-coupled transformers to facilitate the disentanglement of the common features and sensor-specific features. To ensure compatibility among the disentangled features, self-reconstructions of LRMS and PAN images are imposed on them, and source images are reconstructed by self-coupled transformers. In addition to the reconstruction-guided disentangled learning, we maximize the MI between the common features of LRMS and PAN images to improve the correlation of the common features from different images. We also minimize the MI between the common features and sensor-specific features from the same image to reduce the redundancy among them. Through the reconstruction and disentangled representation of source images, sensor-specific features and common features can be decomposed efficiently. Finally, all disentangled features are integrated by a fusion transformer to generate the high spatial resolution multispectral (HRMS) image. Experiments on different datasets demonstrate that the proposed method produces competitive fusion results. The code is available athttps://github.com/RSMagneto/DRFormer.

Existing deep neural network (DNN)-based image fusion methods seldom consider low-rank priors for the decomposition of source images, which cannot efficiently model base and detail components in images. To exploit the low-rank priors better, we propose a deep rank-Ndecomposition network (DRDec-Net) according to the rank-Ndecomposition of source images. Specifically, a rank-Ndecomposition model is first established by imposing low-rank priors on the base component of source images. Then, based on the decomposition model, we construct DRDec-Net, which is composed of low-rank decomposition (LRD) modules, a detail fusion (DetailF) module, and a low-rank fusion (LRF) module. In DRDec-Net, it is assumed that source images share the same base component, which is expressed as the sum of rank-1 components. We employNcascaded LRD modules to extract these rank-1 components from source images. Meanwhile, detail components are obtained by subtracting the base component from source images. Next, the extracted rank-1 components and detail components are integrated by LRF and DetailF modules to produce the base component and detail component of the fused image. Finally, the sum of the two obtained components is regarded as the fused image. Compared to some state-of-the-art methods, experimental results demonstrate that the proposed DRDec-Net can produce a better performance on three image fusion tasks, including infrared and visible images, multi-exposure images, and multi-focus images.

Owing to the efficiency in terms of the modeling of long-range dependencies, transformer-based spectral reconstruction methods have produced satisfactory hyperspectral (HS) images from multispectral (MS) images. Some transformer-based methods applied self-attention to all bands in the HS image to model the relationships among them, which ignore high correlations between adjacent bands and low correlations among nonadjacent ones. To learn the global relationships among all bands and the correlations between adjacent bands simultaneously, this paper proposes a long-short attention network (LSA-Net) for the spectral super-resolution of MS images. Specifically, LSA-Net is composed of cascaded long-range attention blocks and short-range attention blocks. In long-range attention blocks, the transformer is imposed on all channels by modeling each channel as a token. Then, grouped channels are fed into short-range attention blocks for correlation learning, which is inferred from the similarities among neighboring channels. With the introduction of long- and short-range attention, the relationships among spectral bands can be preserved better. Experiments on the CAVE dataset demonstrate the effectiveness of the proposed LSA-Net. The code is available at https://github.com/RSMagneto/LSA-Net.

Recently, pan-sharpening methods based on deep learning (DL) have achieved state-of-the-art results. However, current existing DL-based pan-sharpening methods need to be trained repetitively for different satellite sensors to obtain satisfactory fusion performance and therefore require a large number of training images for each satellite. To deal with these issues, in this paper we propose a unified two-stage spatial and spectral network (UTSN) for pan-sharpening. A branch of networks is constructed for each different satellite, in which the spatial enhancement network (SEN) is shared to improve the spatial details in the fused images from different satellites. A spectral adjustment network (SAN) is employed to capture the spectral characteristics of the specific satellite. Through SAN, the spectral information in the intermediate image from SEN is refined to produce the final fusion results. Such a framework can integrate the datasets from different satellites together for sufficient training of SEN. The proposed method is able to achieve promising pan-sharpening results also for a new satellite with limited training images by only learning a new SAN on the few-shot datasets due to the simple but efficient structure of SAN. The experimental results show that the proposed method can produce state-of-the-art fusion results in both the standard and few-shot cases. The source code is publicly available at https://github.com/RSMagneto/UTSN.

Existing deep neural network (DNN)-based methods have produced good pansharpened images. However, supervised DNN-based pansharpening methods suffer from performance degradation when fusing low spatial resolution multispectral (LR MS) and panchromatic (PAN) images at full resolution. Unsupervised DNN-based methods alleviate the issue, but their training becomes difficult owing to the absence of reference images. This article establishes a novel semi-supervised framework to jointly learn the reconstructions of fused images from reduced- and full-resolution datasets. Specifically, we propose a cross-resolution semi-supervised adversarial learning network (CrossNet), which is composed of a supervised module and an unsupervised module. In these two modules, reduced- or full-resolution source images are disentangled as resolution-invariant components and resolution-aware components by reconstructing the fused images. Moreover, cross-resolution fused images are synthesized to enhance the disentanglement of the two kinds of components. Through the reconstruction of cross-resolution fused images, supervised and unsupervised modules are also coupled efficiently. Then, the semi-supervised framework can simultaneously make use of the supervised information in reduced-resolution datasets and mitigate the performance degradation via full-resolution datasets. Besides, adversarial learning is employed to improve the consistency between resolution-invariant components of source images at different resolutions. Finally, extensive experiments on QuickBird, GeoEye-1, WorldView-2, and WorldView-3 datasets demonstrate that the proposed CrossNet can produce state-of-the-art fusion results in terms of qualitative and quantitative evaluations. The source code is available athttps://github.com/RSMagneto/CrossNet.

Deep unfolding networks have obtained satisfactory performance in the pansharpening task owing to their sufficient interpretability. Inspired by the back-projection (BP) mechanism, we propose a BP-driven model, spatial-spectral dual back-project network (S2DBPN), to fuse the low spatial resolution multispectral (LR MS) and the high spatial resolution panchromatic (PAN) images by exploiting the BP in spatial and spectral domains. Specifically, the proposed S2DBPN is made up of a spatial BP network, a spectral BP network, and a reconstruction network. In the spatial BP network, spatial down- and up-projection modules are derived from BP, which is responsible for the projection of the LR MS image into the spatial domain. By analogy with the spatial BP, we reformulate the degradation between high spatial resolution multispectral (HR MS) and PAN images as spectral down- and up-projections. Then, the spectral BP network is constructed for the projection of the PAN image along the channel dimension. Finally, the features from spatial and spectral BP networks are integrated to produce the desired HR MS image through the reconstruction network. Compared to the state-of-the-art methods, extensive experiments on QuickBird, GeoEye-1, and WorldView-2 datasets demonstrate that our S2DBPN produces better HR MS images in terms of qualitative and quantitative evaluation metrics. The code of S2DBPN is released at: https://github.com/RSMagneto/S2DBPN.

Various deep neural networks (DNNs) have been constructed to inject the spatial information of the panchromatic (PAN) image into the low spatial resolution multispectral (LR MS) image. However, most of them ignore the local dissimilarity (LD) prior between MS and PAN images, which has a negative influence on the fused image. Considering the above-mentioned issues, we propose a deep multiscale local dissimilarity network (DMLD-Net) to learn the LD prior at different scales and enhance the spatial and spectral information in the fused image better. Specifically, we first synthesize a downsampled PAN image from the original PAN image to match the scale of the LR MS image. Then, a LD metric is designed to calculate the dissimilarity map between the two images in feature space. According to the learned dissimilarity map, we utilize a LD-guided attention block (LDGAB) to suppress the impact of LD, which filters out the dissimilar information in the features of the PAN image. To learn the LD prior between MS and PAN images sufficiently, the multiscale architecture is considered and we infer the dissimilar maps hierarchically and inject filtered features into the LR MS image progressively. Finally, the fused image is generated by a reconstruction block. Through the LD learning at different scales, reasonable spatial information is extracted from the PAN image, by which the distortions in the fused image caused by LD can be reduced efficiently. Extensive experiments are conducted on GeoEye-1 and WorldView-2 datasets and the results demonstrate the effectiveness of the proposed DMLD-Net in terms of spatial and spectral preservation. The code is available at https://github.com/RSMagneto/DMLD-Net.

Thanks to their capability of modeling global information, transformers have been recently applied to change detection in remote sensing images. Generally, the changes in terms of shape and appearance of objects lead to relation changes among these objects in multi-temporal images. However, in this context, the attention mechanism in transformers has not been fully explored yet to learn relation changes in the observed scenes. In this paper, we analyze the relation changes in multi-temporal images and propose a cross-temporal difference (CTD) attention to capture these changes efficiently. Through the CTD attention, the changed areas are distinguished better from the unchanged areas. Based on the CTD attention, two CTD-transformer encoders are constructed to extract the features of changed areas from the embedded tokens of multi-temporal images in a cross manner. Then, the extracted features at the coarse scale are further improved to the fine-scale by the corresponding CTD-transformer decoders. In addition, consistency-perception blocks (CPBs) are designed to preserve the structures and contours of changed areas. Finally, all extracted features from multi-temporal images are concatenated to produce the desired change map. Compared to state-of-the-art methods, experimental results on LEVIR-CD, WHU-CD, and CLCD datasets demonstrate that the proposed method produces better performance. The source code is available at https://github.com/RSMagneto/CTD-Former.

In this article, we present a new pansharpening method, a zero-reference generative adversarial network (ZeRGAN), which fuses low spatial resolution multispectral (LR MS) and high spatial resolution panchromatic (PAN) images. In the proposed method, zero-reference indicates that it does not require paired reduced-scale images or unpaired full-scale images for training. To obtain accurate fusion results, we establish an adversarial game between a set of multiscale generators and their corresponding discriminators. Through multiscale generators, the fused high spatial resolution MS (HR MS) images are progressively produced from LR MS and PAN images, while the discriminators aim to distinguish the differences of spatial information between the HR MS images and the PAN images. In other words, the HR MS images are generated from LR MS and PAN images after the optimization of ZeRGAN. Furthermore, we construct a nonreference loss function, including an adversarial loss, spatial and spectral reconstruction losses, a spatial enhancement loss, and an average constancy loss. Through the minimization of the total loss, the spatial details in the HR MS images can be enhanced efficiently. Extensive experiments are implemented on datasets acquired by different satellites. The results demonstrate that the effectiveness of the proposed method compared with the state-of-the-art methods. The source code is publicly available at https://github.com/RSMagneto/ZeRGAN.

Many deep neural networks have been constructed for the pansharpening task. However, the differences between the high and low frequencies in images are not considered in some DNN-based pansharpening methods. As high and low frequencies have different information of images, it is difficult for the same network to learn and reconcile the two kinds of frequencies. Considering the aforementioned differences, we propose a new pansharpening network to fuse the high and low frequencies in low spatial resolution multispectral and panchromatic images separately. Specifically, a high and low frequency fusion network is constructed, which is composed of a high-frequency fusion network and a low-frequency fusion network. In the high-frequency fusion network, skip attention is introduced into U-Net to better retain the high frequencies in feature maps. The low-frequency fusion network uses the involution to capture the dependency among the channels of feature maps. Experiments on the GeoEye-1 dataset reveal that the proposed network outperforms some state-of-the-art methods. The code can be accessed at https://github.com/RSMagneto/HLF-Net.

In this letter, we propose a new unsupervised change detection method based on low-rank prior for multispectral images. It is assumed that the changed and unchanged pixels are from different subspaces due to different appearance and statistical properties. So, low-rank representation (LRR) is employed to find informative pixels from the superpixels of the difference image (DI). Besides, taking the sparsity of changed pixels in the observed scenes into consideration, the selection rule is designed to distinguish these pixels. Then, principal component analysis (PCA) is used for the training of changed and unchanged dictionaries from these pixels. Finally, the change map is estimated by comparing the reconstruction error of each pixel in DI on changed and unchanged dictionaries. By LRR, more representative pixels are found for subsequent dictionary learning, which can efficiently improve the performance of the proposed method. Experiments on multitemporal images from the Landsat satellite demonstrate the effectiveness of the proposed method.

Pan-sharpening methods based on deep neural network (DNN) have produced the state-of-the-art results. However, the common information in the panchromatic (PAN) image and the low spatial resolution multispectral (LRMS) image is not sufficiently explored. As PAN and LRMS images are collected from the same scene, there exists some common information among them, in addition to their respective unique information. The direct concatenation of extracted features leads to some redundancy in the feature space. To reduce the redundancy among features and exploit the global information in source images, we proposed a novel pan-sharpening method by combining the convolution neural network and transformer. Specifically, PAN and LRMS images are encoded as unique features and common features by the subnetworks consisting of convolution blocks and transformer blocks. Then, the common features are averaged and combined with unique features from source images for the reconstruction of the fused image. To extract accurate common features, the equality constraint is imposed on them. Experimental results show that the proposed method outperforms the state-of-the-art methods on both reduced-scale and full-scale datasets. The source code is available athttps://github.com/RSMagneto/TRRNet.

Pansharpening methods based on deep neural networks (DNNs) have been attracting great attention due to their powerful representation capabilities. In this article, to combine the feature maps from different subnetworks efficiently, we propose a novel pansharpening method based on a spatial and spectral extraction network (SSE-Net). Different from the other methods based on DNNs that directly concatenate the features from different subnetworks, we design adaptive feature fusion modules (AFFMs) to merge these features according to their information content. First, the spatial and spectral features are extracted by the subnetworks from low spatial resolution multispectral (LR MS) and panchromatic (PAN) images. Then, by fusing the features at different levels, the desired high spatial resolution MS (HR MS) images are generated by the fusion network consisting of AFFMs. In the fusion network, the features from different subnetworks are integrated adaptively, and the redundancy among them is reduced. Moreover, the spectral ratio loss and the gradient loss are defined to ensure the effective learning of spatial and spectral features. The spectral ratio loss captures the nonlinear relationships among the bands in the MS image to reduce the spectral distortions in the fusion result. Extensive experiments were conducted on QuickBird and GeoEye-1 satellite datasets. Visual and numerical results demonstrate that the proposed method produces better fusion results compared with literature techniques. The source code is available athttps://github.com/RSMagneto/SSE-Net.

In this paper, a low spatial resolution multispectral (LR MS) and panchromatic (PAN) image fusion method based on convolution sparse coding (CSC) is proposed to model the global structures existing in source images. In the proposed method, CSC is adopted to decompose the high frequency (HF) component properly and joint sparse prior is also used to capture the correlation in the bands of MS images. By joint sparsity, the correlation is further inherited into their corresponding feature maps. Then, the spatial information in LR MS image is enhanced well after detailed fusion rule for spatial details. Finally, the fusion image is reconstructed by the fused low frequency and HF. The experimental results on real datasets from QuickBird and Geoeye-1 satellites verify that the proposed method can better preserve the spatial and spectral information in the fused images.

This paper proposed a multispectral (MS) and panchromatic (PAN) image fusion method based on low-rank assumption captured by weighted nuclear norm minimization (WNNM). In this method, low-rank matrix factorization is considered to model the relationship between low spatial resolution (LR) and high spatial resolution (HR) MS images. In the formulation, MS and PAN images are partitioned into patches and then clustered to further guarantee the low-rank property. Besides, WNNM is used to capture the prior about singular values, in which larger singular values are shrunked with smaller weights. By WNNM, the spatial details in MS images can be well enhanced. Finally, the fusion model is established by combining the low-rank matrix factorization with the fidelity term about PAN image. The experimental results on degraded and real datasets demonstrate the effectiveness of the proposed method.