In remote sensing image pansharpening, the fundamental objective is to generate high-resolution multispectral (HRMS) image that preserves spectral integrity while enhancing spatial resolution. However, contemporary approaches lack the ability to perceive the quality of input images, thereby failing to preserve desired performance in complex real-world scenarios characterized by panchromatic (PAN) image degradation (e.g., noise contamination or sensor limitations). To address this problem, we present a novel degradation-aware Taylor expansion approximation (DTEA) network for pansharpening, where DTEA includes the following key procedures: Firstly, the PAN image is hierarchically decomposed into feature maps to represent the degradation information through a proposed Taylor expansion approximation network (TEANet). Next, a multi-level information fusion network (MIFNet) is employed to integrate these feature maps with LRMS images, yielding fused maps with rich spatial and spectral information. Finally, the fused map from each layer is utilized to synthesize the desired HRMS image through inverse Taylor expansion, thereby overcoming diverse information degradation. To rigorously evaluate the effectiveness of our DTEA network, we conduct a systematic performance analysis across PAN images with diverse qualities. Extensive quantitative and qualitative experiments on three datasets demonstrate that our method outperforms state-of-the-art approaches while exhibiting excellent generalization capability in real-world scenarios. Source code will be made publicly available on https://github.com/MysterYxby/DTEA.

High-resolution multispectral (HRMS) image generation through pansharpening requires effective integration of spatial details from panchromatic (PAN) images and spectral information from low-resolution multispectral (LRMS) images. Existing methods often overlook interactions between deep features across different depths and modalities, resulting in spectral distortion and loss of spatial detail. To address this, we propose a dual domain progressive cross-fusion network (D2PCFN) that progressively integrates features in both spatial and frequency domains. The network consists of a dual-branch feature generation module (DBFGM) for deep feature extraction, a dual domain cross-fusion module (D2CFM) for cross-interaction between spatial and frequency representations, and a deep feature reconstruction module (DFRM) for synthesizing high-quality outputs. Extensive experiments on GaoFen-2, QuickBird, WorldView-3, and WorldView-2 datasets demonstrate that our method achieves state-of-the-art accuracy, with average gains of 1.77% in SAM, 1.70% in ERGAS, 0.89% in PSNR, and 1.37% in HQNR over leading methods. Both quantitative and qualitative results confirm the effectiveness and generalization ability of the proposed D2PCFN. Source code will also be shared on https://github.com/MysterYxby/D2PCFN -website link after publication.

It is an extremely challenging task to detect multiscale targets (especially small objects) in remote sensing (RS) images with complex backgrounds. This letter develops a novel RS object detection model, namely dynamic multiscale dilated attention based on YOLOv5 (DMSDA-YOLO), of which the key improvements include: one is that, in the backbone, a multiscale dilated attention fusion module (MDAFM) is proposed to capture multiscale feature information and a coordinate anchor attention (CAA) mechanism is incorporated to increase the focus on target regions while suppressing background interference. The other is that a spatial attention pyramid neck network is proposed to improve its feature fusion capability while a dynamic attention-aware feature extraction module (DAFEM) is introduced to enhance the network’s adaptability to multiscale targets in the neck. Objective and subjective results of experiments on the DIOR, HRRSD, and NWPU VHR-10 datasets demonstrate that our DMSDA-YOLO outperforms existing state-of-the-art object detection approaches in detecting multiscale targets under complex backgrounds, and its competitive computational complexity is beneficial for its extensive application.

Multimodal image registration is a key procedure in remote sensing applications (such as remote sensing image stitching), which faces significant challenges including radiometric discrepancies and local geometric deformations caused by the differences of both sensor and imaging parameters. Traditional methods remove coarse error using global features, making it difficult to identify misregistrations at early stage, thus limiting registration accuracy improvement. When existing convolutional registration neural networks extract deep features, shallow local feature information is usually lost because the network gradually focuses on high-level abstract features, causing local details to be simplified or lost in the global feature construction. Solving this problem will greatly increase the complexity of the model, and the network needs to reorganize and train the data according to specific tasks, which is time-consuming. To address these issues, this letter develops a hybrid registration model with a global-local descriptor. Specifically, we first obtain improved RIFT keypoints via combining rotated and scale invariant corner points produced by the integral scale detection Min-moment with extracted edge points generated by the FAST detection Max-moment. Then, a global-local descriptor is constructed by combining the improved RIFT descriptor with the LoFTR coarse-grained feature descriptor. Finally, a 0–1 distance allocation matrix is formulated to improve the registration success rate (SR). The experimental results show that the proposed method has a powerful capability in improving both generalization and accuracy and outperforms mainstream methods, even the average number of correctly registered correspondences is about two times and 1.7 times higher than LoFTR and RIFT, respectively.

In the image fusion mission, the crucial task is to generate high-quality images for highlighting the key objects while enhancing the scenes to be understood. To complete this task and provide a powerful interpretability as well as a strong generalization ability in producing enjoyable fusion results which are comfortable for vision tasks (such as objects detection and their segmentation), we present a novel interpretable decomposition scheme and develop a target-aware Taylor expansion approximation (T2EA) network for infrared and visible image fusion, where our T2EA includes the following key procedures: Firstly, visible and infrared images are both decomposed into feature maps through a designed Taylor expansion approximation (TEA) network. Then, the Taylor feature maps are hierarchically fused by a dual-branch feature fusion (DBFF) network. Next, the fused map of each layer is contributed to synthesize an enjoyable fusion result by the inverse Taylor expansion. Finally, a segmentation network is jointed to refine the fusion network parameters which can promote the pleasing fusion results to be more suitable for segmenting the objects. To validate the effectiveness of our reported T2EA network, we first discuss the selection of Taylor expansion layers and fusion strategies. Then, both quantitatively and qualitatively experimental results generated by the selected SOTA approaches on three datasets (MSRS, TNO, andLLVIP) are compared in testing, generalization, and target detection and segmentation, demonstrating that our T2EA can produce more competitive fusion results for vision tasks and is more powerful for image adaption. The code will be available at https://github.com/MysterYxby/T2EA.

Taylor expansion is a polynomial for approximating a function constructed by the coefficients of its derivatives at a certain point, where it is a challenging research to utilize the powerful learning ability of deep learning (DL) to characterize the polynomial parts for pursuing its approximate solution. In this article, we develop a cascading residual context-sharing Transformer (RCST) to approximate Taylor expansion for remote sensing (RS) image denoising. Our RCST method includes the following key procedures. First, a mapping function about a latent clean RS image patch is built by employing the low-rank characteristic of its neighborhood RS image blocks, and is expanded into a polynomial with Taylor expansion for its approximate solution. Second, the intrinsic recursive relationship of the neighborhood derivatives is analyzed and is mathematically formulated, which provides a theoretical interpretability for the construction of our RCST model. Third, a lightweight residual network (LRNet) is developed to estimate the base layer, while the RCST is shared to calculate the derivative parts. Finally, to transfer as many rich multiscale details from noisy RS images to estimated results as possible, we adopt a down-/upsampling architecture. Specifically, a spatial-Fourier upsampling (SFUS) operator is reported to preserve both local and global information. Quantitatively and qualitatively experimental results validate that our RCST denoising method can achieve competitive performance and is even superior to other SOTA denoising approaches.

Infrared images are susceptible to thermal radiation. Infrared thermal radiation correction methods based on physical prior may fail while correcting real-world images, because assumed priors do not always hold in the real world, resulting in the presence of thermal radiation residuals. Supervised learning-based methods have the potential to achieve favorable outcomes in the correction of synthetic images. However, due to the unavailability of labeled datasets, their efficacy is limited when applied to real-world images. To address this problem, in this article, to the best of our knowledge, we propose the first semi-supervised learning network for infrared radiation correction in the real world, named SIRCNet. The network is trained using a semi-supervised strategy, which includes a supervised training stage and a self-supervised training stage. In the supervised training stage, we constructed a multilevel wavelet decomposition and reconstruction correction (MWDRC) module for latent image correction and an efficient generalized feature extraction (EGFE) module for bias field estimation. Furthermore, EGFE is composed of one partial channel interactive (PCI) attention block and three effective residual blocks (ERBs). Surface fitting can approximate the thermal radiation bias field of the thermal radiation degradation images. The fit bias field can provide critical prior knowledge that enhances EGFE’s estimation of the thermal radiation bias field. Hence, in the self-supervised training stage, when fine-tuning MWDRC and EGFE using a generator, surface fitting is employed to constrain EGFE. Comparative experiments demonstrate that SIRCNet outperforms existing correction methods on both real and synthetic datasets, achieving the best metrics as well as visualization.

Remote sensing images can accurately display the electromagnetic attribute distribution of various ground objects and can be used in many applications. Nevertheless, stripe noise frequently degrades the quality of the captured images. Most current destriping studies are capable of removing the regular stripe noise. However, their outputs often produce a stripe residual or oversmoothing effect when a complex stripe noise exists. To overcome this limitation, in this article, we propose a variational destriping model based on nonconvex regularization and variable weights. First, our model constrains the stripe sparsity using the normalized$\epsilon $-penalty function and converts it into weighted$\ell _{1}$norm by$\ell _{1}$majorization method. Second, unlike previous models that use scalar smoothing weights to control the output smoothness, we propose to design adaptive vector weights to constrain the smoothness. Moreover, the proposed model introduces the region weight to deal with the extreme regions. In view of the region decomposition technique, we transform our optimization model into a series of 1-D weighted$\ell _{1}$problems in different directions with a linear time solver. Then, an improved alternating direction method is employed to provide fast convergence. The experimental results show that our method achieves a better performance on image detail preservation, complex stripe removal, and a faster convergence than the state-of-the-art methods.

Gradient descent strategy, viewed as an important model optimization method, has been widely used for various tasks (such as model-based image denoising) of computer vision. In the gradient descent denoising model, the learning rate (LR) and residual component are two important parts to be adaptively estimated for its stable point. This letter proposes a deep gradient descent network (DGDNet), including two key points: one is that the LR is designed with eigenvalues of Hessian matrix of remotely sensed images (RSIs) and their local weighted factor (LWF), which can recognize structures from RSIs degraded by additive white Gaussian noise (AWGN). The other is that the residual part is calculated by an U-shaped network (USNet) to speed up the DGDNet convergent to a fixed point. Finally, the two components are plugged into the gradient descent scheme and contribute to an enjoyable result with a few iterations. Quantitatively and qualitatively experimental results demonstrate that the proposed DGDNet can obtain a stable solution efficiently, and produce competitive denoising performance which is even better than that yielded by the state-of-the-art noise reduction methods.

Multi-sensor image registration is a basic and important issue in the field of remote sensing applications. At present, many algorithms have not directly evaluated and analyzed the feature descriptor design of the algorithm. Taking the feature descriptors of RIFT, SIFT, SAR-SIFT and HAPCG as the analysis objects, this paper designs experiments to analyze their stability under gray distortion and local geometric distortion, gives a quantitative evaluation, and reveals the contribution of the feature descriptor of each multi-sensor image registration algorithm in the process of multi-sensor image registration.

Abstract Social networking sites have made photo sharing convenient and, consequently, users of those sites frequently share photos. The proliferation of these social images throughout the Internet has inadvertently exposed an increasing number of personally identifiable information, particularly from the visual information of the faces in the images. Most existing methods to de‐identify faces results in an excessive loss of visual information. To solve this problem, this paper proposes a reliable metrics‐based linear regression model for multilevel privacy measurement of face instances using the size information of face instances. The proposed privacy measurement model provides a novel instance‐level‐based solution to measure privacy levels. The paper also establishes a scientific relationship between the size information of face instances and their privacy levels, quantifying the degree to which face instances need to be de‐identified. Finally, the paper proposes a novel k‐Same‐DT de‐identification method to provide reliable metrics for a linear regression model. It is a real‐time k‐same‐related de‐identification algorithm that combines the PCA dimensionality reduction strategy and the Delaunay triangle‐based face alignment algorithm.The proposed k‐Same‐DT method creates a de‐identified face with a lower identifiable rate and higher structural similarity, and it can provide reliable de‐identification metrics for privacy measurement. Experiments using the classical face dataset demonstrates the effectiveness of the proposed de‐identification method. Extensive experiments and surveys on real‐world social images were also conducted to verify the proposed measurement model.

Remotely sensed images degraded by additive white Gaussian noise (AWGN) are not beneficial for the analysis of their contents. Such a phenomenon is usually modeled as an inverse problem which can be solved by model-based optimization methods or discriminative learning approaches. The former pursue their pleasing performance at the cost of a highly computational burden while the latter are impressive for their fast testing speed but are limited by their application range. To join their merits, this letter proposes a nonlocal self-similar (NSS) block-based deep image denoising scheme, namely deep low-rank prior (DLRP), which includes the following key points: First, the low-rank property of the neighboring NSS patches ordered lexicographically is utilized to model a global objective function (GOF). Second, with the aid of an alternative iteration strategy, the GOF can be easily decomposed into two independent subproblems. One is a quadratic optimization problem, and has a closed-form solution. While the other is a low-rank minimization denoising problem and is learned by deep convolutional neural network (DCNN). Then, the deep denoiser, acted as a modular part, is plugged into the model-based optimization method with adaptive noise level estimation to solve the inverse problem. In the experiments, we first discuss parameter setting and the convergence. Then, quantitative/qualitative comparisons of experimental results validate that the DLRP is a flexible and powerful denoising method to achieve competitive performance which even outperforms those produced by state-of-the-arts.

Low/high or uneven luminance results in low contrast of remotely sensed images (RSIs), which makes it challenging to analyze their contents. In order to improve the contrast and preserving fine weak details of RSIs, this letter proposes a novel enhancement framework to correct luminance guided by weighted least squares (WLS), including the following key parts. First, an image is separated into a base layer and a detail layer by employing the WLS. Then, a learning network is proposed to correct luminance for the base layer enhancement. Next, an enhancement operator for improving the detail layer is computed by using the original image and the enhanced base layer. Finally, the output image is obtained with a fusion of the enhanced base and detail components. Both quantitatively and qualitatively experimental results verify that the proposed method performs better than the state of the arts in contrast improvement and detail preservation.

In the above article[1], the model in(1)should be revised as\begin{equation*}\min _{\mathcal{I}^{\mathcal{B}}}\left\{\left(\mathcal{I}-\mathcal{I}^{\mathcal{B}}\right)^{2}+\lambda\left(a_{x}(\mathcal{I})\left(\frac{\partial \mathcal{I}^{\mathcal{B}}}{\partial x}\right)^{2}+a_{y}(\mathcal{I})\left(\frac{\partial \mathcal{I}^{\mathcal{B}}}{\partial y}\right)^{2}\right)\right\} \end{equation*}to be minimized for${\mathcal {I}}^{\mathcal {B}}$.

Low-quality remotely sensed images (RSIs) are not beneficial for the analysis of many activities including agricultural growth, resident migration, forest fire, and etc. Many previous enhancement schemes improve their quality via changing their illumination. However, these approaches often fail in detail and brightness preservation as well as contrast improvement due to that the information from a single image is limited. To address this issue, an enhancement framework, named as global-local enhancement network (GLE-Net), is proposed to correct the intensity via learning extra information from collected training data, including the following three key steps: first, RSIs are decomposed by the discrete wavelet transformation (DWT) method into the low-frequency component and the detail components. Then, the low-frequency component is improved by the global enhancement network while the detail components are enhanced by the local enhancement network in parallel. Finally, the enhanced components are employed to produce high-quality images with the inverse DWT (IDWT) method. The quantitatively and qualitatively comparable experiments on both synthetic and real-world RSIs validate that the proposed GLE-Net method performs well on preserving brightness and fine details, and even outperforms the state-of-the-arts.

Salient object detection (SOD) task for optical remote sensing images (RSIs) plays an important role in many remote sensing applications. Most of the existing methods train their networks depending on a large amount of pixel-wise datasets. However, such expensive and time-consuming training setting prevents the approaches becoming flexible and scalable solutions. To this end, we explore efficient SOD for optical RSIs based on easily accessible weak supervision source. In this work, we propose a novel end-to-end progressive attention-based feature recovery framework with scribble supervision. Specifically, to better locate challenging salient objects in optical RSIs, an object position module (OPM) is proposed to capture and enhance the long-range semantic dependence of objects’ position information, which depends on the complementary attention mechanism. And to restore the entire salient objects, a context refinement module (CRM) is proposed, which extract local contextual information for better propagating high-level semantics to low-level details. Moreover, to improve the adaptability of the network to the changing scenarios of optical RSIs, we propose a salient region correcting (SRC) mechanism to help the predicted salient regions rectify their saliency values by constraining the saliency relationship between predictions from different augmentation models. In addition, due to the lack of dataset for weakly supervised SOD for optical RSIs, we relabeled an existing large-scale optical RSIs dataset with scribbles, namely EORSSD-S. Experimental results on benchmark datasets demonstrate that the proposed method can outperform other weakly supervised SOD methods. And the proposed method even outperformed some fully supervised methods. https://github.com/melonless/PAFR.

Stripe and random noise are two different degradation phenomena that commonly coexist in optical remote sensing images, and they are often modeled as inverse problems. In model-based inverse problems, analysis and synthesis sparse representations (SSRs) are used as regularization terms to obtain approximate solutions due to their respective merits, i.e., the nonzero coefficients in SSR are usually used to describe an image, while the indexes of zeros in analysis sparse representation (ASR) are used to characterize the stripe. Inspired by these merits, we propose a unified variational framework, called a joint analysis and weighted synthesis (JAWS) sparsity model, to simultaneously separate the clean image and the stripe from a single optical remote sensing image. To solve the JAWS sparsity model efficiently, an alternating minimization optimization strategy is first employed to separate it into two subproblems that are used for different tasks. One called as weighted SSR (WSSR) is the main for optical remote sensing image denoising, which can be effectively solved by employing the weighted singular value thresholding operator, while the other called as ASR is the main approach for optical remote sensing image destriping, which is optimized by adopting the split Bregman iteration. By minimizing the two subproblems alternatively, the proposed JAWS sparsity model is efficiently solved. Finally, both quantitative and qualitative results of experiments on synthetic and real-world optical remote sensing images validate that the proposed approach is effective and even better than the state of the arts.

Non‐local self‐similar patch‐based denoising techniques have been viewed as the most popular denoising approaches in computer vision. This study has proposed a novel iterative weighted sparse representation (IWSR) scheme for X‐ray cardiovascular angiogram image denoising. The main procedures of this scheme include four parts. First, a maximum a posterior (MAP) distribution by the Bayes’ theory is adopted to simultaneously estimate the estimated image and sparse representation with different Gaussian distributions approximating to likelihood prior, non‐local self‐similar patch prior and sparse representation prior. Second, the MAP problem is converted to minimise an energy function using the logarithmic transformation. Third, the function is efficiently solved by the single and effective alternating directions method of multipliers algorithm along with singular value decomposition (SVD) algorithm. Finally, owing to learned dictionary by K‐SVD algorithm, the qualitative and quantitative results of widely synthetic experiments demonstrate that the proposed IWSR denoising method performs effectively and can obtain competitive denoising performance and high‐quality images compared with those advanced denoising methods. The results of extensive experiments on clinical X‐ray angiogram images further illustrate that the IWSR method performs well on noise reduction and vascular structures including edges and capillaries preservation, integral cardiovascular trees of which are beneficial for clinicians to diagnose and analyse cardiovascular diseases.

Enhancement and denoising have always been a pair of conflicting problems in image processing of computer vision. Inspired by an earlier dual-domain filter (DDF), this letter proposes a progressive DDF to simultaneously enhance and denoise low-quality optical remote-sensing images. The main procedure of the proposed enhancement filter has two parts. First, a bilateral filter is exploited as a guide filter to obtain high-contrast images, which are enhanced by a histogram modification method. Then, low-contrast useful structures are restored by a short-time Fourier transform and are enhanced using an adaptive correction parameter. Both the quantitative and qualitative results of experiments on synthetic and real-world low-quality remote-sensing images demonstrate that the proposed method performs well on contrast enhancement, structure preservation, and noise reduction. Moreover, its satisfactory computation time resulting from its simple implementation makes it suitable for extensive application.