Recent studies point to an accuracy gap between humans and Artificial Neural Network (ANN) models when classifying blurred images, with humans outperforming ANNs. To bridge this gap, we introduce a spectral channel-based range-constrained entropy merit function, from which we devise a zero-phase, circular symmetric blind deblurring method. We apply it as a pre-processing step for image classification and test it using pre-trained classification models and images blurred by Gaussian kernels. We compare our method to state-of-the-art restoration methods, showing its superiority, effectively bridging the machine-human gap for most models and blur levels. Our results also rank higher than the competitors in no-reference and full-reference image quality metrics. Notwithstanding the limitation to zero-phase blur, this work shows that, for image pre-processing aimed at visual tasks, it may be advantageous to use merit functions based on vision science and information theory, rather than on the expected error to the latent image.

Abstract. In this work we propose a photorealistic style transfer method for image and video that is based on vision science principles and on a recent mathematical formulation for the deterministic decoupling of sample statistics. The novel aspects of our approach include matching decoupled moments of higher order than in common style transfer approaches, and matching a descriptor of the power spectrum so as to characterize and transfer diffusion effects between source and target, which is something that has not been considered before in the literature. The results are of high visual quality, without spatio-temporal artifacts, and validation tests in the form of observer preference experiments show that our method compares very well with the state of the art. The computational complexity of the algorithm is low, and we propose a numerical implementation that is amenable for real-time video application. Finally, another contribution of our work is to point out that current deep learning approaches for photorealistic style transfer don’t really achieve photorealistic quality outside of limited examples, because the results too often show unacceptable visual artifacts.

We introduce a formalism that justifies and extends a heuristic method for algebraically decoupling deterministic features that recently proved useful for improving feature-based classification. Our new formalism is based on defining transformations inside manifolds, by following trajectories along the features' gradients. Through these transformations we define a normalization that, we demonstrate, allows for decoupling differentiable features. By applying this to sampling moments, we obtain a quasi-analytic solution for the orthokurtosis, a modification of the kurtosis that is not just decoupled from mean and variance, but also from skewness. After theoretically motivating feature decoupling for random data distributions, we illustrate with a regression problem example how decoupled features may perform significantly better than coupled ones.

Feature adjustment, understood as the process aimed at modifying at will global features of given signals, has cardinal importance for several signal processing applications, such as enhancement, restoration, style transfer, and synthesis. Despite of this, it has not yet been approached from a general, theory-grounded, perspective. This work proposes a new conceptual and practical methodology that we term Controlled Feature Adjustment (CFA). CFA provides methods for, given a set of parametric global features (scalar functions of discrete signals), (1) constructing a related set of deterministically decoupled features, and (2) adjusting these new features in a controlled way, i.e., each one independently of the others. We illustrate the application of CFA by devising a spectrally-based hierarchically decoupled feature set and applying it to obtain different types of image synthesis that are not achievable using traditional (coupled) feature sets.

Recently, some techniques for deblurring images having arbitrarily shaped boundaries have been proposed. However, when we observe a blurry foreground (FG) object on a sharp background (BG) there is no abrupt transition between them. Because previous techniques are not designed to cop with a smooth transition area between FG and BG, they avoid it by artificially discarding it from the computations. Here we construct, instead, an observation model that accounts for the occlusion photometric effect of the FG object on the BG, and we illustrate its realism by comparing a simulation with a real capture. Then we use that observation model to pose a Bayesian MAP estimation problem with an L2-relaxed L0 sparse prior and two Gaussian likelihood terms (for the noise, and for the BG interference in the smooth transition area), which we solve by alternating optimizations. Using simulations, we demonstrate a higher performance of our method, compared to two state-of-the-art unconstrained boundary restoration techniques.

Global features of signals are usually coupled, in the sense that not every combination of features' values is mathematically possible. The existence of forbidden combinations complicates the topology of the joint feature vector set, and precludes an interpretation of features as independent descriptors. Another consequence is that there is no easy and direct way of imposing a feature to the signal without perturbing others. Therefore, signal analysis, processing and synthesis applications based on global descriptors often resent from feature coupling. Here we first propose a sequential method for modifying arbitrary features independently of each other. Then we derive a decoupling method based on hierarchical normalization that yields new features with gradients mutually orthogonal everywhere in the signal space. Here we apply this method to the first ten marginal moments, emerging new sample statistics that, as we show, significantly improve texture classification.

In recent years, boundary problems associated to image restoration have attracted the attention of imaging scientists. We explore here the case of arbitrary boundary shapes (like blurred-background images), starting from a general discussion and advancing towards realist conditions, with simulations and real photographic images. We describe our Spectral-Pre-Adaptation (SPA) method, and compare it to the successful unconstrained boundary conditions Alternating Direction Method of Multipliers (UBC ADMM) method. Preliminary results indicate that SPA, combined with efficient restoration algorithms, such as Constrained Dynamic L2-relaxed L0, may set the new state-of-the-art, in performance and computational terms, for this kind of problems.

We address the problem of finding the optimal focus of an optical system with spherical aberration, under three optimization criteria: the classical optical root-mean-square second order moment minimization, the expected mean square error on the sensor, and the expected mean square error at the output of a Wiener restoration filter. We observe that these three criteria may behave very differently, and, particularly, the classical optical criterion typically provides very poor results in comparison. This has a direct impact on the design of new hybrid optical-digital imaging systems, which account for the image quality after an optics-aware embedded digital restoration stage. We present some very encouraging results for simulations under different noise and aberration levels.

Here we propose an efficient estimation method to interpolate new samples in a blurred and aliased observation, in such a way that (1) aliasing artifacts in an ulterior restoration are mitigated, and (2) thanks to aliasing we may recover some spatial frequencies beyond the Nyquist frequency (super-resolution). The only requirement is having a good approximation of the blurring kernel in high resolution (HR). The method consists of two sequential steps: (1) to perform a Maximum Likelihood Interpolation, according to a Gaussian model; and (2) to apply a standard deconvolution to the interpolated image, using the HR blurring kernel in both steps. Simulations show strong improvements with respect to first deconvolving the unprocessed observation and then doing a standard interpolation with splines, and also with respect to first doing a standard interpolation and then deconvolving.

We propose a method to simulate the rich, scale-dependent dynamics of water waves. Our method preserves the dispersion properties of real waves, yet it supports interactions with obstacles and is computationally efficient. Fundamentally, it computes wave accelerations by way of applying a dispersion kernel as a spatially variant filter, which we are able to compute efficiently using two core technical contributions. First, we design novel, accurate, and compact pyramid kernels which compensate for low-frequency truncation errors. Second, we design a shadowed convolution operation that efficiently accounts for obstacle interactions by modulating the application of the dispersion kernel. We demonstrate a wide range of behaviors, which include capillary waves, gravity waves, and interactions with static and dynamic obstacles, all from within a single simulation.

We present a simple model for characterizing and synthesizing binary texture, whose main appeal is that it introduces the concept of reverse analysis, directly inspired by Quantum Computation. This approach allows to overcome the need of a synthesis algorithm, in the classical sense of the term. Instead, we compute, for every possible N×N pattern (N small), a set of M texture parameters. By selecting some texture parameters' values in the transformed space, we can readily identify the set of local patterns compatible with those parameters (reverse analysis). Then, we sample the local texture pattern by randomly choosing among them, same as if we were observing entangled qubit patterns. To provide visual continuity of adjacent patterns, we force them to match their pixels in overlapping areas. Although the texture model presented here is extremely simple (M = 3), results are promising, and the same concept could be applied to more sophisticated models.

We propose a novel formulation for relaxed analysis-based sparsity in multiple dictionaries as a general type of prior for images, and apply it for Bayesian estimation in image restoration problems. Our formulation of a ℓ2-relaxed ℓ0 pseudo-norm prior allows for an especially simple maximum a posteriori estimation iterative marginal optimization algorithm, whose convergence we prove. We achieve a significant speedup over the direct (static) solution by using dynamically evolving parameters through the estimation loop. As an added heuristic twist, we fix in advance the number of iterations, and then empirically optimize the involved parameters according to two performance benchmarks. The resulting constrained dynamic method is not just fast and effective, it is also highly robust and flexible. First, it is able to provide an outstanding tradeoff between computational load and performance, in visual and objective, mean square error and structural similarity terms, for a large variety of degradation tests, using the same set of parameter values for all tests. Second, the performance benchmark can be easily adapted to specific types of degradation, image classes, and even performance criteria. Third, it allows for using simultaneously several dictionaries with complementary features. This unique combination makes ours a highly practical deconvolution method.

Directly applying circular de-convolution to real-world blurred images usually results in boundary artifacts. Classic boundary extension techniques fail to provide likely results, in terms of a circular boundary-condition observation model. Boundary reflection gives raise to non-smooth features, especially when oblique oriented features encounter the image boundaries. Tapering the boundaries of the image support, or similar strategies (like constrained diffusion), provides smoothness on the toroidal support; however this does not guarantee consistency with the spectral properties of the blur (in particular, to its zeros). Here we propose a simple, yet effective, model-derived method for extending real-world blurred images, so that they become likely in terms of a Gaussian circular boundary-condition observation model. We achieve artifact-free results, even under highly unfavorable conditions, when other methods fail.

Recently we proposed an efficient technique based on analysis-based sparsity in tight frames to restore images affected by shift-invariant blur and additive white Gaussian noise. Here we apply the same alternate marginal optimization idea used in that work, but dealing with combined Poissonian-Gaussian noise, which we approximate as Gaussian, additive and signal-dependent. We operate (1) by adding to the original image and its associated sparse vector another auxiliary variable to be optimized in the loop: the blurred, but not yet noisy, image; and (2) by using a quadratic soft constraint. We also re-formulate the previous prior modelling in order to perform a standard maximum a posteriori estimation, and generalize the approach to allow for a multi-frame prior. The proposed technique is computationally efficient and yields state-of-the-art performance.

Recently we proposed an efficient technique based on analysis-based sparsity in tight frames to restore images affected by linear blur and additive white Gaussian noise. Such technique performed a joint optimization of a cost function in the image and coefficient space by alternating their corresponding marginal minimizations, until convergence. Here we propose, first, a more standard Bayesian (MAP) re-formulation of the model/method, yielding the same cost function. Second, we propose a heuristical twist in the method, consisting of running only a few iterations (too few to reach convergence), with a dynamic shrinkage plan obtained by maximizing a performance measure in a set of deblurring tests. Third, we propose a multi-frame prior scheme allowing for different thresholds in different frames. Compared to the original method, we achieve a very important speed-up, while keeping high robustness and state-of-the-art performance.

Sparse optimization in overcomplete frames has been widely applied in recent years to ill-conditioned inverse problems. In particular, analysis-based sparse optimization consists of achieving a certain trade-off between fidelity to the observation and sparsity in a given linear representation, typically measured by some ¿pquasi-norm. Whereas most popular choice for p is 1 (convex optimization case), there is an increasing evidence on both the computational feasibility and higher performance potential of non-convex approaches (0 ¿ p < 1). The extreme p = 0 case is especial, because analysis coefficients of typical images obtained using typical pyramidal frames are not strictly sparse, but rather compressible. Here we model the analysis coefficients as a strictly sparse vector plus a Gaussian correction term. This statistical formulation allows for an elegant iterated marginal optimization. We also show that it provides state-of-the-art performance, in a least-squares error sense, in standard deconvolution tests.

The local statistical properties of photographic images, when represented in a multi-scale basis, have been described using Gaussian scale mixtures (GSMs). In that model, each spatial neighborhood of coefficients is described as a Gaussian random vector modulated by a random hidden positive scaling variable. Here, we introduce a more powerful model in which neighborhoods of each subband are described as a finite mixture of GSMs. We develop methods to learn the mixing densities and covariance matrices associated with each of the GSM components from a single image, and show that this process naturally segments the image into regions of similar content. The model parameters can also be learned in the presence of additive Gaussian noise, and the resulting fitted model may be used as a prior for Bayesian noise removal. Simulations demonstrate this model substantially outperforms the original GSM model.

We propose a new formulation to the sparse approximation problem for the case of tight frames which allows to minimize the cost function using gradient descent. We obtain a generalized version of the iterative hard thresholding (IHT) algorithm, which provides locally optimal solutions. In addition, to avoid non-favorable minima we use an annealing technique consisting of gradually de-smoothing a previously smoothed version of the cost function. This results in decreasing the threshold through the iterations, as some authors have already proposed as a heuristic. We have adapted and applied our method to restore images having localized information losses, such as missing pixels. We present high-performance in-painting results.

In recent years, Bayes least squares-Gaussian scale mixtures (BLS-GSM) has emerged as one of the most powerful methods for image restoration. Its strength relies on providing a simple and, yet, very effective local statistical description of oriented pyramid coefficient neighborhoods via a GSM vector. This can be viewed as a fine adaptation of the model to the signal variance at each scale, orientation, and spatial location. Here, we present an enhancement of the model by introducing a coarser adaptation level, where a larger neighborhood is used to estimate the local signal covariance within every subband. We formulate our model as a BLS estimator using space-variant GSM. The model can be also applied to image deconvolution, by first performing a global blur compensation, and then doing local adaptive denoising. We demonstrate through simulations that the proposed method, besides being model-based and noniterative, it is also robust and efficient. Its performance, measured visually and in L2-norm terms, is significantly higher than the original BLS-GSM method, both for denoising and deconvolution.

In a previous work, we presented an extension of the original Bayes least squares-Gaussian scale mixtures (BLS-GSM) denoising algorithm that also compensated the blur. However, that method had some problems: a) it could not compensate for some blurring kernels; b) its performance depended critically on having an accurate estimation of the original power spectral density (PSD); and c) it could not be easily adapted to a spatially variant description of the image statistics. In this work we propose a two-step restoration method that overcomes these problems by first performing a global blur image compensation, and then applying a spatially adaptive local denoising, in an overcomplete pyramid. Our method is efficient, robust and non-iterative. We demonstrate through simulations that it provides state-of-the-art performance.

We present a simple and robust method for finding sparse representations in overcomplete transforms, based on minimization of the L0-norm. Our method is better than current solutions based on minimization of the L1-norm in terms of energy compaction. These results strongly question the equivalence of minimizing both norms in real conditions. We also show application to in-painting (interpolation of lost pixels).

We describe an adaptive denoising method for images decomposed in overcomplete oriented pyramids. Our approach integrates two kinds of adaptation: 1) a 'coarse' adaptation, where a large window is used within each subband to estimate the local signal covariance; 2) a 'fine' adaptation, which uses small neighborhoods of coefficients modelled as the product of a Gaussian and a hidden multiplier, i.e., as Gaussian scale mixtures (GSM). The former provides adaptation to local spectral features, whereas the latter adapts to local energy fluctuations. We formulate our method as a Bayes least squares estimator using spatially variant GSMs. We also discuss the importance of image representation, compare the results using two different representations with complementary features, and study the effect of merging their results. We demonstrate through simulation that our method surpasses the state-of-the-art performance, in a L/sub 2/-norm sense.

We present an efficient linear minimum mean square error (LMMSE) method for reconstructing full color images from single sensor color filter array (CFA) data. We use a representative set of full color images to estimate the joint spatial-chromatic covariance among pixel color components. Then, we derive from it a set of joint color-space, small linear kernels which predict the missing color samples as linear combinations of their neighbor observed samples. The color arrangement of the local mosaic varies with the window's location, and this results into a different predictor for every local mosaic and color sample. As an extension, we include blur and noise in the training process, obtaining localized mosaic-constrained Wiener estimators that partially compensate for these degradations. We show that this simple method provides an excellent trade-off between performance and computational cost.

We describe an efficient generalized expectation maximization algorithm for estimating the spectral features of a noise source corrupting an observed image. We use a statistical model for images decomposed in an overcomplete oriented pyramid. Each neighborhood of clean pyramid coefficients is modeled as a Gaussian scale mixture, whereas the noise is assumed Gaussian. Combining this GEM technique with a previous Bayesian denoise estimator, we obtain a full blind denoising algorithm, able to deal with homogeneous, Gaussian or mesokurtotic, noise sources of arbitrary covariance. Results demonstrate the high performance of the method for a wide range of corruption sources.

A statistical model for images decomposed in an overcomplete wavelet pyramid is described. Each neighborhood of pyramid coefficients is modeled as the product of a Gaussian vector of known covariance, and an independent hidden positive scalar random variable. We propose an efficient Bayesian estimator for the pyramid coefficients of an image degraded by linear distortion (e.g., blur) and additive Gaussian noise. We demonstrate the quality of our results in simulations over a wide range of blur and noise levels.

We describe a method for removing noise from digital images, based on a statistical model of the coefficients of an overcomplete multiscale oriented basis. Neighborhoods of coefficients at adjacent positions and scales are modeled as the product of two independent random variables: a Gaussian vector and a hidden positive scalar multiplier. The latter modulates the local variance of the coefficients in the neighborhood, and is thus able to account for the empirically observed correlation between the coefficient amplitudes. Under this model, the Bayesian least squares estimate of each coefficient reduces to a weighted average of the local linear estimates over all possible values of the hidden multiplier variable. We demonstrate through simulations with images contaminated by additive white Gaussian noise that the performance of this method substantially surpasses that of previously published methods, both visually and in terms of mean squared error.

We describe a statistical model for images decomposed in an overcomplete wavelet pyramid. Each coefficient of the pyramid is modeled as the product of two independent random variables: an element of a Gaussian random field, and a hidden multiplier with a marginal log-normal prior. The latter modulates the local variance of the coefficients. We assume subband coefficients are contaminated with additive Gaussian noise of known covariance, and compute a MAP estimate of each multiplier variable based on observation of a local neighborhood of coefficients. Conditioned on this multiplier, we then estimate the subband coefficients with a local Wiener estimator. Unlike previous approaches, we (a) empirically motivate our choice for the prior on the multiplier; (b) use the full covariance of signal and noise in the estimation; (c) include adjacent scales in the conditioning neighborhood. To our knowledge, the results are the best in the literature, both visually and in terms of squared error.

We describe a novel method of removing additive white noise of known variance from photographic images. The method is based on a characterization of the statistical properties of natural images represented in a complex wavelet decomposition. Specifically, we decompose the noisy image into wavelet subbands, estimate the autocorrelation of both the noise-free raw coefficients and their magnitudes within each subband, impose these statistics by projecting onto the space of images having the desired autocorrelations, and reconstruct an image from the modified wavelet coefficients. This process is applied repeatedly, and can be accelerated to produce optimal results in only a few iterations. Denoising results compare favorably to three reference methods, both perceptually and in terms of mean squared error.

A common method for texture representation is to use the marginal probability densities over the outputs of a set of multi-orientation, multi-scale filters as a description of the texture. We propose a technique, based on independent component analysis, for choosing the set of filters that yield the most informative marginals, meaning that the product over the marginals most closely approximates the joint probability density function of the filter outputs. The algorithm is implemented using a steerable filter space. Experiments involving both texture classification and synthesis show that compared to principal components analysis, ICA provides superior performance for modeling of natural and synthetic textures.

We present a parametric statistical characterization of texture images in the context of an overcomplete complex wavelet frame. The characterization consists of the local autocorrelation of the coefficients in each subband, the local autocorrelation of the coefficent magnitudes, and the cross-correlation of coefficient magnitudes at all orientations and adjacent spatial scales. We develop an efficient algorithm for sampling from an implicit probability density conforming to these statistics, and demonstrate its effectiveness in synthesizing artificial and natural texture images.