Synthetic aperture radar (SAR) and optical images provide complementary imaging information, and their joint application holds broad prospects in military reconnaissance and aircraft visual navigation, with the key task of achieving accurate image matching. However, due to significant differences in imaging characteristics and the presence of strong noise interference in complex electromagnetic environments, SAR imaging may face serious challenges in matching with optical images. In addition, limited hardware resources on airborne platforms make it difficult for existing matching algorithms to meet the requirements for online real-time matching. In this article, a lightweight, high-performance, and robust method for robust SAR and optical image matching is proposed. The main contributions include the design of a lightweight convolutional neural network (CNN) and an SAR image random mask noise-resistant feature reconstruction network (named RMSO-ConvNeXt). First, a lightweight pseudo-Siamese dense feature extraction network module was explored for pixel-wise feature extraction of SAR and optical images, which can efficiently extract common features between heterogeneous images. Second, an SAR random noise mask feature reconstruction network was designed to address noise interference in SAR images, and a new joint loss function was constructed to significantly enhance the model’s ability to resist strong noise interference. Furthermore, a high-quality large-scale SAR and optical dataset has been made publicly available to contribute to the advancement and application of image matching. Finally, extensive experiments demonstrate that compared to current state-of-the-art matching methods, the proposed approach exhibits stronger robustness, better matching performance, and smaller model parameters. The code and dataset are shared onhttps://github.com/yeyuanxin110/RMSO-ConvNeXt.

Due to the complementary nature of optical and SAR images, their alignment is of increasing interest. However, due to the significant radiometric differences between them, precise matching becomes a very challenging problem. Although current advanced structural features and deep learning-based methods have proposed feasible solutions, there is still much potential for improvement. In this paper, we propose a hybrid matching method using attention-enhanced structural features (namely AESF), which combines the advantages of both handcrafted-based and learning-based methods to improve the accuracy of optical and SAR image matching. It mainly consists of two modules: a novel effective multi-branch global attention (MBGA) module and a joint multi-cropping image matching loss function (MCTM) module. The MBGA module is designed to focus on shared information in structural feature descriptors of heterogeneous images across space and channel dimensions, significantly improving the expressive capacity of the classical structural features and generating more refined and robust image features. The MCTM module is constructed to fully exploit the association between global and local information of the input image, which can optimize the triple loss discriminator to discriminate positive and negative samples. To validate the effectiveness of the proposed method, it is compared with five state-of-the-art matching methods by using various optical and SAR datasets. The experimental results show that the matching accuracy at the 1-pixel threshold is improved by about 1.8%-8.7% compared with the most advanced deep learning method (OSMNet) and 6.5%-23% compared with the handcrafted description method (CFOG).

Identifying feature correspondences between multimodal images is facing enormous challenges because of the significant differences both in radiation and geometry. To address these problems, we propose a novel feature matching method (named R2FD2) that is robust to radiation and rotation differences, which consists of a repeatable feature detector and a rotation-invariant feature descriptor. In the first stage, a repeatable feature detector called the Multi-channel Auto-correlation of the Log-Gabor (MALG) is presented for feature detection, which combines the multi-channel auto-correlation strategy with the Log-Gabor wavelets to detect interest points (IPs) with high repeatability and uniform distribution. In the second stage, a rotation-invariant feature descriptor is constructed, named the Rotation-invariant Maximum index map of the Log-Gabor (RMLG), which includes fast assignment of dominant orientation and construction of feature representation. In the process of fast assignment of dominant orientation, a Rotation-invariant Maximum Index Map (RMIM) is built to address rotation deformations. Then, the proposed RMLG incorporates the rotation-invariant RMIM with the spatial configuration of DAISY to improve RMLG’s resistance to radiation and rotation variances. Finally, we conduct experiments to validate the matching performance of our R2FD2utilizing different types of multimodal image datasets. Experimental results show that the proposed R2FD2outperforms five state-of-the-art feature matching methods. Moreover, our R2FD2achieves the accuracy of matching within two pixels and has a great advantage in matching efficiency over contrastive methods.

Fusion of Synthetic aperture radar (SAR) and optical images is a significant topic in the field of remote sensing. As a typical category of image fusion methods, pixel level image fusion algorithms have been widely used in SAR-optical image fusion to integrate their complementary information and facilitate the subsequent interpretation and application. The effectiveness of these methods has been demonstrated in different literatures based on the experiment carried on specific, individual datasets, which make a comprehensive comparison of these algorithms difficult to achieve. This paper builds a sub-meter SAR and optical image dataset covering different types of scenes, the performance of 11 pixel level image methods is then investigated based on qualitative and quantitative analysis. Result shows the gradient pyramid (GP) achieve a high quality fusion when dealing with Optical-SAR image fusion task of residents, the non subsampled contourlet transform (NSCT) performs best when fusing images containing farmland and mountains.

Automatic and precise matching between optical and synthetic aperture radar (SAR) images is still a challenging task because of significant radiation and texture differences between such images. Recently, structure feature-based methods are popular for the matching of SAR and optical images. However, current structure descriptors include many noninformative features, which degrade their matching performance. To address that, we present a robust matching method by a multiscale masked structure feature representation. We first extract pixelwise gradient structure features on multiple scales of images. Then, a mask is constructed according to large contours of an image, which is used to increase the contribution of the main structure region and alleviate the influence of noninformative regions. Finally, a fast template scheme based on fast Fourier transform (FFT) is employed to obtain correspondences. The proposed method is tested using the optical and SAR images from the Sentinel and GaoFen sensors. Experiment results show that the proposed method significantly improves the matching performance compared with the state-of-the-art methods, especially for the images with poor structure features.

Registration for multisensor or multimodal image pairs with a large degree of distortions is a fundamental task for many remote sensing applications. To achieve accurate and low-cost remote sensing image registration, we propose a multiscale framework with unsupervised learning, named MU-Net. Without costly ground truth labels, MU-Net directly learns the end-to-end mapping from the image pairs to their transformation parameters. MU-Net stacks several deep neural network (DNN) models on multiple scales to generate a coarse-to-fine registration pipeline, which prevents the backpropagation from falling into a local extremum and resists significant image distortions. We design a novel loss function paradigm based on structural similarity, which makes MU-Net suitable for various types of multimodal images. MU-Net is compared with traditional feature-based and area-based methods, as well as supervised and other unsupervised learning methods on the optical-optical, optical-infrared, optical-synthetic aperture radar (SAR), and optical-map datasets. Experimental results show that MU-Net achieves more comprehensive and accurate registration performance between these image pairs with geometric and radiometric distortions. We share the code implemented by Pytorch athttps://github.com/yeyuanxin110/MU-Net.