Plug-and-Play (PnP) algorithms leverage the power of modern denoisers for image reconstruction. They have been shown to deliver state-of-the-art reconstructions using CNN denoisers. It was established in recent works that convergence of these iterative algorithms can be guaranteed using nonexpansive denoisers. However, integrating nonexpansivity into gradient-based learning is challenging. Existing algorithms for training nonexpansive denoisers often cannot guarantee nonexpansivity or are computationally intensive. The present work is based on the observation that the convergence of PnPFBS and PnP-BBS (PnP based on Forward-Backward and BackwardBackward Splittings) can be guaranteed using contractive denoisers. In this regard, we show that by unfolding FBS iterations applied to wavelet denoising, we can construct contractive image denoisers whose regularization capacity is comparable to CNN denoisers. To the best of knowledge, this is the first work to introduce a simple framework for training denoisers that are provably contractive.

Hyperspectral (HS) images have high spectral but low spatial resolutions, while multispectral (MS) images, on the other hand, have low spectral but high spatial resolutions. In HS–MS fusion, the HS and MS images are combined to obtain a single image with high spatial and spectral resolutions. Such images are typically textured and exhibit repetitive structures. To exploit this prior, we propose a nonlocal weighted total-variation regularizer. The novelty of the design is that the following hold: 1) we use a weighted norm, where the weights are derived from the MS image and 2) pixel variations over nonlocal neighborhoods are considered. We incorporate the regularizer into a standard convex optimization framework involving quadratic data-fidelity terms. We develop an efficient ADMM algorithm for solving this optimization problem—the novelty in this regard is that we use a variable splitting technique that results in the closed-form solutions of the ADMM subproblems. We report results on standard datasets demonstrating that the proposed regularizer can recover fine textures (as opposed to local pixel-based methods) and outperform the state-of-the-art methods.

There has been tremendous research on the design of image regularizers over the years, from simple Tikhonov and Laplacian to sophisticated sparsity and CNN-based regularizers. Coupled with a model-based loss function, these are typically used for image reconstruction within an optimization framework. The technical challenge is to develop a regularizer that can accurately model realistic images and be optimized efficiently along with the loss function. Motivated by the recent plug-and-play paradigm for image regularization, we construct a quadratic regularizer whose reconstruction capability is competitive with state-of-the-art regularizers. The novelty of the regularizer is that, unlike classical regularizers, the quadratic objective function is derived from the observed data. Since the regularizer is quadratic, we can reduce the optimization to solving a linear system for applications such as superresolution, deblurring, inpainting, etc. In particular, we show that using iterative Krylov solvers, we can converge to the solution in few iterations, where each iteration requires an application of the forward operator and a linear denoiser. The surprising finding is that we can get close to deep learning methods in terms of reconstruction quality. To the best of our knowledge, the possibility of achieving near state-of-the-art performance using a linear solver is novel.

We propose a fast algorithm for an adaptive variant of the classical bilateral filter, where the range kernel is allowed to vary from pixel to pixel. Several fast and accurate algorithms have been proposed for bilateral filtering, but they assume that the same range kernel is used at each pixel and hence cannot be used for adaptive bilateral filtering (ABF). Only recently, it was shown that fast algorithms for ABF can be developed by approximating the local histogram around each pixel using polynomials. The present algorithm is derived using an entirely different approximation, namely, the range kernels across all pixels are jointly approximated (compressed) using singular value decomposition (SVD). The SVD involves a very large matrix and cannot be computed exactly; however, we are able to get a sufficiently accurate approximation using the Nyström method (without populating/storing the entire matrix). We show that this SVD-type decomposition allows us to approximate the adaptive bilateral filter using fast convolutions. To demonstrate the speed and accuracy of the proposed algorithm in relation to existing algorithms, we use it for texture filtering, JPEG deblocking, and detail enhancement.

We consider the problem of synthetically fusing a high-spatial, low-spectral resolution image with a low-spatial, high-spectral resolution image to achieve high spatial and spectral resolution. In practice, the images to be fused are usually misaligned and need to be registered before fusion is carried out. However, due to significant difference in spatial resolutions (between the input images), it can be difficult to register them accurately. We consider a variational framework for simultaneous registration and fusion along with regularization that is built upon a standard observation (forward) model. Using a mix of alternating minimization and proximal gradient descent, we obtain an algorithm in which we iteratively optimize over rotations/translations, the model mismatch, and the regularization term. Motivated by the “plug-and-play” paradigm for image restoration, we propose to replace (i) the alignment process by an efficient registration method, and (ii) the proximal map (of the regularizer) with a powerful denoiser. As the iterations proceed, we notice that better registration and regularization results in improved fusion. We demonstrate that our method is competitive with state-of-the-art fusion algorithms on standard datasets, and is particularly effective even for larger misalignments.

In hyperspectral image fusion, a high resolution multispectral (MS) image is combined with a low resolution hyperspectral (HS) image to obtain a high resolution HS image. In this work, we propose a "plug-and-play" framework for HS-MS fusion, where the inversion step at each iteration involves the solution of a linear system, and the regularization is performed using a high-dimensional kernel denoiser. The core contribution is the design of the denoiser, which can denoise an HS-image at low complexity using clustering and convolutions. In particular, it can exploit the inter-band correlations, which cannot be done using band-by-band denoising. An important technical aspect of our denoiser is that it can be expressed as the proximal map of a proper, closed, and convex regularizer, which guarantees the convergence of the plug-and-play iterations. Preliminary results suggest that we are competitive with state-of-the-art algorithms for HS-MS fusion in terms of speed and restoration accuracy.

The bilateral and nonlocal means filters are instances of kernel-based filters that are popularly used in image processing. It was recently shown that fast and accurate bilateral filtering of grayscale images can be performed using a low-rank approximation of the kernel matrix. More specifically, based on the eigendecomposition of the kernel matrix, the overall filtering was approximated using spatial convolutions, for which efficient algorithms are available. Unfortunately, this technique cannot be scaled to high-dimensional data such as color and hyperspectral images. This is simply because one needs to compute/store a large matrix and perform its eigendecomposition in this case. We show how this problem can be solved using the Nyström method, which is generally used for approximating the eigendecomposition of large matrices. The resulting algorithm can also be used for nonlocal means filtering. We demonstrate the effectiveness of our proposal for bilateral and nonlocal means filtering of color and hyperspectral images. In particular, our method is shown to be competitive with state-of-the-art fast algorithms, and moreover, it comes with a theoretical guarantee on the approximation error.

Existing fast algorithms for bilateral and nonlocal means filtering mostly work with grayscale images. They cannot easily be extended to high-dimensional data such as color and hyperspectral images, patch-based data, flow-fields, etc. In this paper, we propose a fast algorithm for high-dimensional bilateral and nonlocal means filtering. Unlike existing approaches, where the focus is on approximating the data (using quantization) or the filter kernel (via analytic expansions), we locally approximate the kernel using weighted and shifted copies of a Gaussian, where the weights and shifts are inferred from the data. The algorithm emerging from the proposed approximation essentially involves clustering and fast convolutions, and is easy to implement. Moreover, a variant of our algorithm comes with a guarantee (bound) on the approximation error, which is not enjoyed by existing algorithms.We present some results for high-dimensional bilateral and nonlocal means filtering to demonstrate the speed and accuracy of our proposal. Moreover, we also show that our algorithm can outperform state-of-the-art fast approximations in terms of accuracy and timing.

We consider the problem of approximating a truncated Gaussian kernel using Fourier (trigonometric) functions. The computation-intensive bilateral filter can be expressed using fast convolutions by applying such an approximation to its range kernel, where the truncation in question is the dynamic range of the input image. The error from such an approximation depends on the period, the number of sinusoids, and the coefficient of each sinusoid. For a fixed period, we recently proposed a model for optimizing the coefficients using least squares fitting. Following the compressive bilateral filter (CBF), we demonstrate that the approximation can be improved by taking the period into account during the optimization. The accuracy of the resulting filtering is found to be at least as good as the CBF, but significantly better for certain cases. The proposed approximation can also be used for non-Gaussian kernels, and it comes with guarantees on the filtering accuracy.

Several useful algorithms for image filtering involve non-linear processing of high-dimensional data. Instances of these so-called high-dimensional filters are the bilateral, joint-bilateral, and non-local means filters. Real-time implementation of high-dimensional filters can be challenging. In this paper, we present a simple and fast algorithm for generic high-dimensional filtering. The algorithm is based on a linearization mechanism, which allows us to approximate the high-dimensional filtering using a series of spatial convolutions. We use clustering for the linearization, whereby we are able to exploit the strong correlation between the components of the high-dimensional image. The highlight of our method is that we can prove that the approximation error (the gap between the fast approximation and the exact filtering) vanishes with the increase in the number of clusters. To the best of our knowledge, this is the first algorithm for high-dimensional filtering that enjoys this theoretical guarantee. In fact, we provide empirical evidence which suggests that this basic requirement is not met by the state-of-the-art Adaptive Manifolds algorithm. We use the proposed algorithm for edge-preserving smoothing and denoising of color and hyperspectral images. The results demonstrate that our algorithm is competitive with existing fast algorithms.

The overall goal of our research is to create a low-cost, home-based diagnostic and rehabilitation tool to assist the rapid functional recovery of people with upper limb (UL) dysfunction (due to muscular dystrophy and stroke)-the diagnostic aspect of which is discussed in this paper. Viewed in the broader context of robot-assisted neurorehabilitation, the proposed framework takes advantage of manipulation of an instrumented manipulandum along prescribed movement patterns to create a sensitive, quantitative, computer-based assessment/diagnostic tool. Specifically, in this paper, we discuss various aspects of an implementation using a COTS force-feedback driving wheel, interfaced to a PC to create a virtual driving environment (VDE). Coupled with the exercise/movement protocols (structured as driving exercises along paths of varying complexity), this results in creation of a low-cost, user-friendly and immersive personal-movement trainer that is well suited for deployment in patients' homes. Our initial analysis of data collected from 5 healthy subjects indicates that the tool has adequate sensitivity/resolution to distinguish even between healthy subjects and shows considerable promise for diagnosis in diseased population (which is currently underway).