Abstract. The classical Mumford–Shah (MS) model has been successful in some medical image segmentation tasks, providing segmentation results with smooth boundaries of objects. However, the MS model, which operates at the pixel level of the images, faces challenges when dealing with medical images with low contrast or unclear edges. In this paper, we begin by using a feature extractor to capture high-dimensional deep features that contain more comprehensive semantic information than pixel-level data alone. Inspired by the MS model, we develop a variational model that incorporates threshold dynamics (TD) regularization for segmenting each feature. We obtain the final segmentation result for the original image by assembling segmentation results of all the features. This process results in MS-MGNet, a lightweight trainable segmentation network with a similar architecture to many encoder–decoder networks. The intermediate layers of MS-MGNet are designed by unrolling the numerical scheme based on the multigrid method for solving the variational model. We provide interpretability for the encoder–decoder architecture by elucidating the roles of each layer and offering explanations of the underlying mathematical models. By incorporating the TD regularizer, we integrate spatial priors from the variational models into the network architecture, resulting in better segmentation results with smoother edges and a certain robustness to noise. Compared to some relevant methods, experimental results on the selected data sets with low contrast or unclear edges show that the proposed method can achieve better segmentation performance with fewer parameters, even when trained on smaller data sets.

Image segmentation is a crucial but challenging task that has many applications. In medical imaging, for instance, intensity inhomogeneity and noise are common. In thigh muscle images, different muscles are closely packed together and there are often no clear boundaries between them. Intensity based segmentation models cannot separate one muscle from another. To solve such problems, in this work we present a segmentation model with adaptive spatial priors from joint registration. This model combines segmentation and registration in a unified framework to leverage their positive mutual influence. The segmentation is based on a modified Gaussian mixture model, which integrates intensity inhomogeneity and spatial smoothness. The registration plays the role of providing a shape prior. We adopt a modified sum of squared difference fidelity term and Tikhonov regularity term for registration and also utilize a Gaussian pyramid and parametric method for robustness. The connection between segmentation and registration is guaranteed by the cross entropy metric that aims to make the segmentation map (from segmentation) and deformed atlas (from registration) as similar as possible. This joint framework is implemented within a constraint optimization framework, which leads to an efficient algorithm. We evaluate our proposed model on synthetic and thigh muscle MR images. Numerical results show the improvement as compared to segmentation and registration performed separately and other joint models.

Hyperspectral image (HSI) super-resolution refers to enhancing the spatial resolution of a 3-D image with many spectral bands (slices). It is a seriously ill-posed problem when the low-resolution (LR) HSI is the only input. It is better solved by fusing the LR HSI with a high-resolution (HR) multispectral image (MSI) for a 3-D image with both high spectral and spatial resolution. In this article, we propose a novel nonnegative and nonlocal 4-D tensor dictionary learning-based HSI super-resolution model using group-block sparsity. By grouping similar 3-D image cubes into clusters and then conduct super-resolution cluster by cluster using 4-D tensor structure, we not only preserve the structure but also achieve sparsity within the cluster due to the collection of similar cubes. We use 4-D tensor Tucker decomposition and impose nonnegative constraints on the dictionaries and group-block sparsity. Numerous experiments demonstrate that the proposed model outperforms many state-of-the-art HSI super-resolution methods.

Pansharpening is an important application in remote sensing image processing. It can increase the spatial-resolution of a multispectral image by fusing it with a high spatial-resolution panchromatic image in the same scene, which brings great favor for subsequent processing such as recognition, detection, etc. In this paper, we propose a continuous modeling and sparse optimization based method for the fusion of a panchromatic image and a multispectral image. The proposed model is mainly based on reproducing kernel Hilbert space (RKHS) and approximated Heaviside function (AHF). In addition, we also propose a Toeplitz sparse term for representing the correlation of adjacent bands. The model is convex and solved by the alternating direction method of multipliers which guarantees the convergence of the proposed method. Extensive experiments on many real datasets collected by different sensors demonstrate the effectiveness of the proposed technique as compared with several state-of-the-art pansharpening approaches.

In this paper, we propose a continuous modeling and sparse optimization based method for the fusion of a panchromatic (PAN) image and a multispectral (MS) image. The proposed model is mainly based on reproducing kernel Hilbert space (RKHS) and approximated Heaviside function (AHF). In addition, we also design an iterative strategy to recover more image details. The final model is a convex one and solved by the designed alternating direction method of multipliers (ADMM) which guarantees the convergence of the proposed method. Experimental results on two real datasets corresponding to different sensors and different resolutions demonstrate the effectiveness of the proposed approach as compared with several state-of-the-art pansharpening approaches.

Image super-resolution (SR), a process to enhance image resolution, has important applications in satellite imaging, high-definition television, medical imaging, and so on. Many existing approaches use multiple low-resolution (LR) images to recover one high-resolution (HR) image. In this paper, we present an iterative scheme to solve single-image SR problems. It recovers a high-quality HR image from solely one LR image without using a training data set. We solve the problem from image intensity function estimation perspective and assume that the image contains smooth and edge components. We model the smooth components of an image using a thin-plate reproducing kernel Hilbert space and the edges using approximated Heaviside functions. The proposed method is applied to image patches, aiming to reduce computation and storage. Visual and quantitative comparisons with some competitive approaches show the effectiveness of the proposed method.

In this paper, we propose an automatic and efficient method to enhance edge sharpness of images. Starting from an image with blur edges, we improve the edges using transformed Heaviside functions for better visualization. In addition, we provide an efficient method to directly compute the scaling and shifting factors of the transformed Heaviside functions, so that blur edges can be improved accurately. Experimental results show that the proposed method is fast and can get sharper image edges than some recent state-of-the-art edge enhancement methods. We also apply the edge sharpening method to image super-resolution and obtained promising results.

It is a challenging task to reconstruct images from their noisy, blurry, and/or incomplete measurements, especially those with important details and features such as medical magnetic resonance (MR) and CT images. We propose a novel regularization model that integrates two recently developed regularization tools: total generalized variation (TGV) by Bredies, Kunisch, and Pock; and shearlet transform by Labate, Lim, Kutyniok, and Weiss. The proposed model recovers both edges and fine details of images much better than the existing regularization models based on the total variation (TV) and wavelets. Specifically, while TV preserves sharp edges but suffers from oil painting artifacts, TGV “selectively regularizes” different image regions at different levels and thus largely avoids oil painting artifacts. Unlike the wavelet transform, which represents isotropic image features much more sparsely than anisotropic ones, the shearlet transform can efficiently represent anisotropic features such as edges, curves, and so on. The proposed model based on TGV and the shearlet transform has been tested in the compressive sensing context and produced high-quality images using fewer measurements than the state-of-the-art methods. The proposed model is solved by splitting variables and applying the alternating direction method of multiplier (ADMM). For certain sensing operators, including the partial Fourier transform, all the ADMM subproblems have closed-form solutions. Convergence of the algorithm is briefly mentioned. The numerical simulations presented in this paper use the incomplete Fourier, discrete cosine, and discrete wavelet measurements of MR images and natural images. The experimental results demonstrate that the proposed regularizer preserves various image features (including edges and textures), much better than the TV/wavelet based methods.

We propose EdgeCS---an edge guided compressive sensing reconstruction approach---to recover images of higher quality from fewer measurements than the current methods. Edges are important image features that are used in various ways in image recovery, analysis, and understanding. In compressive sensing, the sparsity of image edges has been successfully utilized to recover images. However, edge detectors have not been used on compressive sensing measurements to improve the edge recovery and subsequently the image recovery. This motivates us to propose EdgeCS, which alternatively performs edge detection and image reconstruction in a mutually beneficial way. The edge detector of EdgeCS is designed to faithfully return partial edges from intermediate image reconstructions even though these reconstructions may still have noise and artifacts. For complex-valued images, it incorporates joint sparsity between the real and imaginary components. EdgeCS has been implemented with both isotropic and anisotropic discretizations of total variation and tested on incomplete $k$-space (spectral Fourier) samples. It applies to other types of measurements as well. Experimental results on large-scale real/complex-valued phantom and magnetic resonance (MR) images show that EdgeCS is fast and returns high-quality images. For example, it exactly recovers the $256 \times 256$ Shepp--Logan phantom from merely 7 radial lines ($3.03\%$ $k$-space), which is impossible for most existing algorithms. It is able to accurately reconstruct a $512 \times 512$ MR image with 0.05 white noise from $20.87\%$ radial samples. On complex-valued MR images, it obtains recoveries with faithful phases, which are important in many medical applications. Each of these tests took around 30 seconds on a standard PC. Finally, the algorithm is GPU friendly.

Compressive sensing (CS) reconstructs images from a small number of projections. We propose EdgeCS - edge guided CS reconstruction - to recover images of higher qualities from fewer measurements than the current state-of-the-art methods. Accurate edge information can significantly improve image recovery quality and speed, but such information is encoded in the CS measurements of an image. To take advantage of edge information in CS recovery, EdgeCS alternatively performs CS reconstruction and edge detection in a way that each benefits from the latest solution of the other. EdgeCS is fast and returns high-quality images. It exactly recovers the 256 × 256 Shepp-Logan phantom from merely 7 radial lines (or 3.03% k-space), which is impossible for most existing algorithms. It accurately reconstructs a 512 × 512 magnetic resonance image from 21% noisy samples. Moreover, it is also able to reconstruct complex-valued images. Each took about 30 seconds on an ordinary laptop. The algorithm can be easily ported to GPUs for a speedup of more than 10 folds.

Piecewise constant and piecewise smooth Mumford-Shah (MS) models have been widely studied and used for image segmentation. More complicated than piecewise constant MS, global Gaussian intensity distribution within each partitioned region has also been studied. However, all these frameworks are limited in power and robustness in finding objects whose interiors have high noise level and/or complex multi-modal intensity distribution. To overcome these drawbacks,we propose a non-parametric kernel based model which is able to simultaneously segment and smooth images without adding extra smoothing terms. At each point within each partitioned smooth region, a Gaussian kernel with mean the intensity of the given to-be-segmented image at this point and a small local variance depending on the location is applied to create a nonparametric intensity statistics approximation. To save computation, a quadratic kernel involving simple calculation could replace the Gaussian kernel that involves expensive exponential calculation. We demonstrate the superiority of proposed model over other models by showing segmentation results from various images with different levels and types of noise.