Image smoothing is a fundamental technique in image processing, designed to eliminate perturbations and textures while preserving dominant structures. It plays a pivotal role in numerous high-level computer vision tasks. More recently, both traditional and deep learning-based smoothing methods have been developed. However, existing algorithms frequently encounter issues such as gradient reversals and halo artifacts. Furthermore, the smoothing strength of deep learning-based models, once trained, cannot be adjusted for adapting different complexity levels of textures. These limitations stem from the inability of previous approaches to achieve an optimal balance between smoothing intensity and edge preservation. Consequently, image smoothing while maintaining edge integrity remains a significant challenge. To address these challenges, we propose a novel edge-aware smoothing model that leverages a relative wavelet domain representation. Specifically, by employing wavelet transformation, we introduce a new measure, termed Relative Wavelet Domain Representation (RWDR), which effectively distinguishes between textures and structures. Additionally, we present an innovative edge-aware scale map that is incorporated into the adaptive bilateral filter, facilitating mutual guidance in the smoothing process. This paper provides complete theoretical derivations for solving the proposed non-convex optimization model. Extensive experiments substantiate that our method has a competitive superiority with previous algorithms in edge-preserving and artifact removal. Visual and numerical comparisons further validate the effectiveness and efficiency of our approach in several applications of image smoothing.

Plug-and-play (PnP) methods with deep denoisers have shown impressive results in imaging problems. They typically require strong convexity or smoothness of the fidelity term and a (residual) non-expansive denoiser for convergence. These assumptions, however, are violated in Poisson inverse problems, and non-expansiveness can hinder denoising performance. To address these challenges, we propose a cocoercive conservative (CoCo) denoiser, which may be (residual) expansive, leading to improved denoising performance. By leveraging the generalized Helmholtz decomposition, we introduce a novel training strategy that combines Hamiltonian regularization to promote conservativeness and spectral regularization to ensure cocoerciveness. We prove that CoCo denoiser is a proximal operator of a weakly convex function, enabling a restoration model with an implicit weakly convex prior. The global convergence of PnP methods to a stationary point of this restoration model is established. Extensive experimental results demonstrate that our approach outperforms closely related methods in both visual quality and quantitative metrics.

There are two main methods that can be used to accelerate MRI reconstruction: parallel imaging and compressed sensing. To further accelerate the sampling process, the combination of these two methods has been extensively studied in recent years. However, existing MRI reconstruction methods often overlook the exploration of high-frequency information of images, leading to sub-optimal recovery of fine details in the reconstructed results. To address this issue, we conduct an in-depth analysis of image gradients and propose a novel MRI reconstruction model based on Maximum a Posteriori (MAP) estimation. We first establish the Cumulative Deviation from Maximum Gradient magnitude (CDMG) prior for fully sampled MR images through theoretical analysis, then incorporate this explicit CDMG prior along with an implicit deep prior to form the prior probability term. This combination of priors strikes a balance between physically informed constraints and data-driven adaptability, aiding in the recovery of meaningful high-frequency information. Additionally, we introduce a multi-order gradient operator to enhance the observation model, thereby improving the accuracy of the likelihood term. Through MAP estimation, we develop a novel accelerated MRI reconstruction model, the optimization of which is achieved by unrolling it into a convolutional neural network structure, referred to as DDGU-Net. Extensive experimental results demonstrate the effectiveness of our approach in reconstructing high-quality MR images and achieving state-of-the-art (SOTA) results, particularly at higher acceleration factors.

Abstract. Blind image deblurring is one of the most critical issues in digital image processing. The main goal of blind image deblurring is to estimate the blur kernel and the intermediate image with a blurry image as input. In this paper, we propose a new algorithm for blind image deblurring based on the image smoothing priors. We notice that the salient edge is significant in estimating the blur kernel. If we can find a strategy that can preserve the salient edges of images and smooth out unnecessary details in the deblurring process, the estimated blur kernel will be more accurate. According to this observation, we draw on the experience of image smoothing and propose a new model based on the smoothing priors. For binary images, we extend our model by considering binary constraints. We also extend our method to the nonuniform deblurring problem. Numerically, we use the half-quadratic splitting method to minimize the optimization problem. We also propose a new template-based interpolated algorithm to solve the [Formula: see text] minimization problem. In the experiment part, we test our method on various datasets to show its effectiveness. Compared with other related methods, our method can estimate blur kernels more accurately and generate fewer artifacts.

The development of sensor technology has resulted in the accumulation of extensive spatiotemporal traffic information, which holds great potential for predicting traffic patterns and improving traffic management strategies. Nevertheless, dealing with missing data poses a significant challenge for the intelligent traffic system (ITS). To address this issue, this study employs a nonconvex smoothly clipped absolute deviation (SCAD) penalty customized for tensors to surrogate tensor rank and incorporates the deep plug-and-play (PnP) prior into the low-rank tensor completion (LRTC) model. An efficient iterative framework is formulated to integrate these penalties into the alternating direction method of multipliers (ADMM) method. Moreover, two imputation methods, namely LRTC-SCAD and LRTC-SCAD-DeepPnP, are developed, affirming that the LRTC-SCAD method ensures convergence to the global optimum. We conduct simulated experiments using real-world traffic datasets, and our proposed methods outperform state-of-the-art imputation methods. For instance, on the Portland dataset, LRTC-SCAD achieves a noteworthy 9.86% improvement in mean absolute percentage error (MAPE) compared to the cutting-edge LRTC method while consuming only 28.26% of its total running time. Similarly, on the PeMS dataset, LRTC-SCAD-DeepPnP achieves an average 11.59% enhancement in MAPE, with visually compelling improvements in imputation results, further validating its efficacy in maintaining local consistency. The code is available athttps://github.com/peterchen96/LRTC_DeepPnP.

The score-based generative model (SGM) can generate high-quality samples, which have been successfully adopted for magnetic resonance imaging (MRI) reconstruction. However, the recent SGMs may take thousands of steps to generate a high-quality image. Besides, SGMs neglect to exploit the redundancy in space. To overcome the above two drawbacks, in this article, we propose a fast and reliable SGM (FRSGM). First, we propose deep ensemble denoisers (DEDs) consisting of SGM and the deep denoiser, which are used to solve the proximal problem of the implicit regularization term. Second, we propose a spatially adaptive self-consistency (SASC) term as the regularization term of the -space data. We use the alternating direction method of multipliers (ADMM) algorithm to solve the minimization model of compressed sensing (CS)-MRI incorporating the image prior term and the SASC term, which is significantly faster than the related works based on SGM. Meanwhile, we can prove that the iterating sequence of the proposed algorithm has a unique fixed point. In addition, the DED and the SASC term can significantly improve the generalization ability of the algorithm. The features mentioned above make our algorithm reliable, including the fixed-point convergence guarantee, the exploitation of the space, and the powerful generalization ability.

Magnetic Resonance Imaging (MRI) plays a crucial role in diagnosing and treating various diseases. However, the long acquisition time of MRI scans often leads to patient discomfort and motion artifacts. Consequently, accelerating MRI speed is essential. Researchers have combined Deep Learning with Compressed Sensing and Parallel Imaging to advance MRI. However, many existing methods fail to effectively recover the fine details and structures in Magnetic Resonance images. To address these challenges, we propose a novel model for accelerated parallel MRI reconstruction. Our model incorporates a high-frequency fidelity method into the reconstruction process, explicitly emphasizing the recovery of high-frequency information. Additionally, we consider the joint priori distribution between the reconstructed images from each coil. Using the variable splitting approach, the proposed model is unrolled as an end-to-end network termed HFF-Net. Experimental results demonstrate that our method outperforms state-of-the-art techniques, yielding high-quality MR images with enhanced detail and fine structure recovery.

Deep denoisers have shown excellent performance in solving inverse problems in signal and image processing. In order to guarantee the convergence, the denoiser needs to satisfy some Lipschitz conditions like non-expansiveness. However, enforcing such constraints inevitably compromises recovery performance. This paper introduces a novel training strategy that enforces a weaker constraint on the deep denoiser called pseudo-contractiveness. By studying the spectrum of the Jacobian matrix, relationships between different denoiser assumptions are revealed. Effective algorithms based on gradient descent and Ishikawa process are derived, and further assumptions of strict pseudo-contractiveness yield efficient algorithms using half-quadratic splitting and forward-backward splitting. The proposed algorithms theoretically converge strongly to a fixed point. A training strategy based on holomorphic transformation and functional calculi is proposed to enforce the pseudo-contractive denoiser assumption. Extensive experiments demonstrate superior performance of the pseudo-contractive denoiser compared to related denoisers. The proposed methods are competitive in terms of visual effects and quantitative values.

Abstract. Image decolorization is widely used in single-channel image processing, black-and-white printing, etc. Decolorization aims to generate a perceptually satisfactory gray image that preserves the contrast of the color image. It is known that histogram equalization can enhance the global image contrast by effectively spreading out the most frequent intensity values. Meanwhile, local contrast features such as salient edges and local details have large local variances, which can be enhanced by maximizing local variance. Inspired by these facts, we propose a two-stage decolorization method based on histogram equalization and local variance maximization. In the first stage, we assume that the decolorized gray image is a linear combination of the three channels of the color image, and the combination coefficients are three global weights. Then we propose a constrained variational histogram equalization model to optimize the global weights. The resulting gray image has good global contrast. To further enhance the local contrast, in the second stage, we use local weight combination to express the color image and maximize the local variance by forcing the local weights to be close to the global weights. Numerically, the global weights can be estimated by a gradient-based solver or a discrete searching solver, and the local weights are solved by an iterative solver. Theoretically, we discuss the properties of the energy functions and the convergence of the algorithm. Our proposed method better preserves global and local contrast than state-of-the-art decolorization algorithms.

Abstract Recently, deep convolutional neural networks have been successfully used for image denoising due to their favourable performance. This paper examines the error feedback mechanism to image denoising and propose an error feedback denoising network. Specifically, we use the down‐and‐up projection sequence to estimate the noise feature. By the residual connection, the clean structures are removed from the noise features. The essential difference between the proposed network and other existing feedback networks is the projection sequence. Our error feedback projection sequence is down‐and‐up, which is more suitable for image denoising than the existing up‐and‐down order. Moreover, we design a compression block to improve the expression ability of the general 11 convolutional compression layer. The advantage of our well‐designed down‐and‐up block is that the network parameters are fewer than other feedback networks and the receptive field is enlarged. We have implemented our error feedback denoising network on denoising and JPEG image deblocking. Extensive experiments verify the effectiveness of our down‐and‐up block and demonstrate that our error feedback denoising network is comparable with the state‐of‐the‐art. The code will be open source. The source codes for reproducing the results can be found at: https://github.com/Houruizhi/EFDN.

Low-light image enhancement based on deep convolutional neural networks (CNNs) has revealed prominent performance in recent years. However, it is still a challenging task since the underexposed regions and details are always imperceptible. Moreover, deep learning models are always accompanied by complex structures and enormous computational burden, which hinders their deployment on mobile devices. To remedy these issues, in this paper, we present a lightweight and efficient Luminance-aware Pyramid Network (LPNet) to reconstruct normal-light images in a coarse-to-fine strategy. The architecture is comprised of two coarse feature extraction branches and a luminance-aware refinement branch with an auxiliary subnet learning the luminance map of the input and target images. Besides, we propose a multi-scale contrast feature block (MSCFB) that involves channel split, channel shuffle strategies, and contrast attention mechanism. MSCFB is the essential component of our network, which achieves an excellent balance between image quality and model size. In this way, our method can not only brighten up low-light images with rich details and high contrast but also significantly ameliorate the execution speed. Extensive experiments demonstrate that our LPNet outperforms state-of-the-art methods both qualitatively and quantitatively.

In this paper, we propose a novel Retinex-based fractional-order variational model for severely low-light images. The proposed method is more flexible in controlling the regularization extent than the existing integer-order regularization methods. Specifically, we decompose directly in the image domain and perform the fractional-order gradient total variation regularization on both the reflectance component and the illumination component to get more appropriate estimated results. The merits of the proposed method are as follows: 1) small-magnitude details are maintained in the estimated reflectance. 2) illumination components are effectively removed from the estimated reflectance. 3) the estimated illumination is more likely piecewise smooth. We compare the proposed method with other closely related Retinex-based methods. Experimental results demonstrate the effectiveness of the proposed method.

In this study, the authors propose a new loss function for denoising convolutional neural network (DnCNN) for salt‐and‐pepper noise (SPN). Based on the motivation of utilising the mask of SPN, firstly from the usual SPN‐denoising restoration equation, the authors establish a perfect restoration condition; the restored image is precisely the clean image if this condition holds. Then they design a mask‐involved loss function to encourage the network to satisfy this condition in training progress. Experimental results demonstrate that compared with general DnCNN and other state‐of‐the‐art SPN denoising methods, DnCNN equipped with the proposed loss function involving mask (MaskDnCNN) is more effective, robust and efficient.

Pansharpening is a process of acquiring a multi-spectral image with high spatial resolution by fusing a low resolution multi-spectral image with a corresponding high resolution panchromatic image. In this paper, a new pansharpening method based on the Bayesian theory is proposed. The algorithm is mainly based on three assumptions: 1) the geometric information contained in the pan-sharpened image is coincident with that contained in the panchromatic image; 2) the pan-sharpened image and the original multi-spectral image should share the same spectral information; and 3) in each pan-sharpened image channel, the neighboring pixels not around the edges are similar. We build our posterior probability model according to above-mentioned assumptions and solve it by the alternating direction method of multipliers. The experiments at reduced and full resolution show that the proposed method outperforms the other state-of-the-art pansharpening methods. Besides, we verify that the new algorithm is effective in preserving spectral and spatial information with high reliability. Further experiments also show that the proposed method can be successfully extended to hyper-spectral image fusion.

In this paper, we propose and develop a novel nonlocal variational technique based on structural similarity (SS) information for image restoration problems. In the literature, patches extracted from images are compared according to their pixel values, and then nonlocal filtering can be employed for image restoration. The disadvantage of this approach is that intensity-based patch distance may not be effective in image restoration, especially for images containing texture or structural information. The main aim of this paper is to propose using SS between image patches to develop nonlocal regularization models. In particular, two types of nonlocal regularizing functions are studied: an SS-based nonlocal quadratic function (SS-NLH1) and an SS-based nonlocal total variation function (SS-NLTV) for regularization of image restoration problems. Moreover, we employ iterative algorithms to solve these SS-NLH1 and SS-NLTV variational models numerically and discuss the convergence of these algorithms. The experimental results are presented to demonstrate the effectiveness of the proposed models.

A variant of fuzzy c‐means (FCM) clustering algorithm for image segmentation is provided. Unlike the ‐norm distance in FCM, with norm is used to measure the distance of the pixel intensity to its cluster centre in the energy functional. Moreover, local spatial information and colour information are incorporated into the model to enhance the robustness to noise and outliers. The proposed algorithm is called fuzzy local information (FLILp) clustering. To overcome the difficulty of finding cluster centres, ‐norm distance is approximated by weighted distance. The advantages of FLILp are: (i) it is strongly robust to noise and outliers, (ii) it is applied to the original image and (iii) it preserves image edges. Numerical examples and comparisons of image segmentation on both synthetic and real images illustrate the outstanding performance and robustness of the proposed method.

Abstract In this paper, we extend the use of the partial differential equation (PDE) method to head visualization with computed tomography (CT) data and show how the two primary medical visualization means, surface reconstruction, and volume rendering can be integrated into one single framework through PDEs. Our scheme first performs head segmentation from CT slices using a variational approach, the output of which can be readily used for extraction of a small set of PDE boundary conditions. With the extracted boundary conditions, head surface reconstruction is then executed. Because only a few slices are used, our method can perform head surface reconstruction more efficiently in both computational time and storage cost than the widely used marching cubes algorithm. By elaborately introducing a third parameterwto the PDE method, a solid head can be created, based on which the head volume is subsequently rendered with 3D texture mapping. Instead of designing a transfer function, we associate the alpha value of texels of the 3D texture with the PDE parameterwthrough a linear transform. This association enables the production of a visually translucent head volume. The experimental results demonstrate the feasibility of the developed head visualization method. Copyright © 2015 John Wiley & Sons, Ltd.

In this paper, we focus on variational approaches for image inpainting in transform domain and propose two new algorithms, iterative coupled transform domain inpainting (ICTDI) and iterative decoupled transform domain inpainting. In the derivation of ICTDI, we use operator splitting and the quadratic penalty technique to get a new approximate problem of the basic model. By the alternating minimization method, the approximate problem can be decomposed as three relatively simple subproblems with closed-form solutions. However, ICTDI is not efficient when some adaptive regularization operator is used, such as the learned BM3D frame. To overcome this drawback, with some modifications, we decouple our framework into three relatively independent parts: denoising, linear combination in the transform domain, and linear combination in the image domain. Therefore, we can use any existing denoising method in the denoising step. We consider three choices for regularization operators in our approach: gradient operator, tight framelet transform, and learned BM3D frame. The numerical experiments and comparisons on various images demonstrate the effectiveness of the proposed methods. The convergence of the numerical algorithms is proved under some assumptions.

In this paper, we propose a new fast variational approach to dehaze and denoise simultaneously. The proposed method first estimates a transmission map using a windows adaptive method based on the celebrated dark channel prior. This transmission map can significantly reduce the edge artifact in the resulting image and enhance the estimation precision. The transmission map is then converted to a depth map, with which the new variational model can be built to seek the final haze- and noise-free image. The existence and uniqueness of a minimizer of the proposed variational model is further discussed. A numerical procedure based on the Chambolle--Pock algorithm is given, and the convergence of the algorithm is ensured. Extensive experimental results on real scenes demonstrate that our method can restore vivid and contrastive haze- and noise-free images effectively.

Color-to-grayscale conversion is the process used to convert a color image to a grayscale one, which is a basic tool in digital printing, photograph rendering, and single-channel image processing. The main aim of this paper is to propose a variational approach for image decolorization by variance maximization. Our idea is to use an energy functional to determine local transformations for combining red, green, and blue channel pixel values together by maximizing the local variance of the output grayscale image and preserving the brightness of the input color image. In order to minimize the differences among the local transformations at nearby pixel locations, the total variation regularization of the transformation is incorporated into the functional for the decolorization process. The existence and uniqueness of the minimizer of the variational model can be shown. We also present an effective algorithm for solving the variational model numerically, and show the convergence of the algorithm. Experimental results are reported to demonstrate the effectiveness of the proposed method, and its performance is better than those of the other testing methods for a set of benchmark color images.

In this paper, we extend an existing universal variational framework for image inpainting with new numerical algorithms. Given certain regularization operator Φ and denoting u the latent image, the basic model is to minimize the l(p), (p=0,1) norm of Φu preserving the pixel values outside the inpainting region. Utilizing the operator splitting technique, the original problem can be approximated by a new problem with extra variable. With the alternating minimization method, the new problem can be decomposed as two subproblems with exact solutions. There are many choices for Φ in our approach such as gradient operator, wavelet transform, framelet transform, or other tight frames. Moreover, with slight modification, we can decouple our framework into two relatively independent parts: 1) denoising and 2) linear combination. Therefore, we can take any denoising method, including BM3D filter in the denoising step. The numerical experiments on various image inpainting tasks, such as scratch and text removal, randomly missing pixel filling, and block completion, clearly demonstrate the super performance of the proposed methods. Furthermore, the theoretical convergence of the proposed algorithms is proved.

Pan-sharpening is a process of acquiring a high resolution multispectral (MS) image by combining a low resolution MS image with a corresponding high resolution panchromatic (PAN) image. In this paper, we propose a new variational pan-sharpening method based on three basic assumptions: 1) the gradient of PAN image could be a linear combination of those of the pan-sharpened image bands; 2) the upsampled low resolution MS image could be a degraded form of the pan-sharpened image; and 3) the gradient in the spectrum direction of pan-sharpened image should be approximated to those of the upsampled low resolution MS image. An energy functional, whose minimizer is related to the best pan-sharpened result, is built based on these assumptions. We discuss the existence of minimizer of our energy and describe the numerical procedure based on the split Bregman algorithm. To verify the effectiveness of our method, we qualitatively and quantitatively compare it with some state-of-the-art schemes using QuickBird and IKONOS data. Particularly, we classify the existing quantitative measures into four categories and choose two representatives in each category for more reasonable quantitative evaluation. The results demonstrate the effectiveness and stability of our method in terms of the related evaluation benchmarks. Besides, the computation efficiency comparison with other variational methods also shows that our method is remarkable.

The goal of this letter is to provide an elliptical filter to improve image coherence for the task of image smoothing and inpainting. The kernel of this filter is adaptively weighted and its shape is determined by local coherence estimation. The long axis of its ellipse is the same as the coherence direction and we put more weight there to enhance coherence. Compared with the related anisotropic partial differential equations (PDEs) or wavelet shrinkage methods, the proposed filter is extremely simple, instinctive and easy to code. Numerical examples and comparisons illustrate clearly the good performance of the proposed filter.

The Aubert–Aujol (AA) model is a variational method for multiplicative noise removal. In this paper, we study some basic properties of the regularization parameter in the AA model. We develop a method for automatically choosing the regularization parameter in the multiplicative noise removal process. In particular, we employ spatially varying regularization parameters in the AA model in order to restore more texture details of the denoised image. Experimental results are presented to demonstrate that the spatially varying regularization parameters method can obtain better denoised images than the other tested multiplicative noise removal methods.

The goal of this paper is to develop a multiphase image segmentation method based on fuzzy region competition. A new variational functional with constraints is proposed by introducing fuzzy membership functions which represent several different regions in an image. The existence of a minimizer of this functional is established. We propose three methods for handling the constraints of membership functions in the minimization. We also add auxiliary variables to approximate the membership functions in the functional such that Chambolle's fast dual projection method can be used. An alternate minimization method can be employed to find the solution, in which the region parameters and the membership functions have closed form solutions. Numerical examples using grayscale and color images are given to demonstrate the effectiveness of the proposed methods.