Empirically, it has been observed that adding momentum to Stochastic Gradient Descent (SGD) accelerates the convergence of the algorithm. However, the literature has been rather pessimistic, even in the case of convex functions, about the possibility of theoretically proving this observation. We investigate the possibility of obtaining accelerated convergence of the Stochastic Nesterov Accelerated Gradient (SNAG), a momentum-based version of SGD, when minimizing a sum of functions in a convex setting. We demonstrate that the average correlation between gradients allows to verify the strong growth condition, which is the key ingredient to obtain acceleration with SNAG. Numerical experiments, both in linear regression and deep neural network optimization, confirm in practice our theoretical results.

Abstract. To address the problem of separating images into a structure and a texture component, we introduce a joint structure-texture model. Instead of considering two separate regularizations for each component, we consider a joint structure-texture model regularization function that takes both components as inputs. This allows for the regularization to take into account the shared information between the two components. We present evidence that shows a performance gain compared to separate regularization models. To implement the joint regularization, we adapt the plug-and-play framework to our setting, using deep neural networks. We train the corresponding deep prior on a randomly generated synthetic dataset of examples of this model. In the context of image decomposition, we show that while trained on synthetic datasets, our plug-and-play method generalizes well to natural images. Furthermore, we show that this framework permits us to leverage the structure-texture decompositions to solve inverse imaging problems such as inpainting.

We consider the problem of recovering off-the-grid spikes from linear measurements in the context of Single Molecule Localization Microscopy (SMLM). State of the art model-based methods such as Over-Parametrized Continuous Orthogonal Matching Pursuit (OP-COMP) with Projected Gradient Descent (PGD) have been shown to successfully recover those signals. The computational cost of these methods scales linearly with the number of measurements. When this number of measurements is large with respect to the dimensionality of the signal, we propose to reduce it with a so-called sketching operator. Based on recent results on compressive sensing in the space of measures, we approximate the ideal sketching operator (benefiting from theoretical recovery guarantees), in the context of SMLM. This sketching method coupled to OP-COMP with PGD shows significant improvements in calculation time in realistic synthetic microscopy experiments.

The expected patch log-likelihood algorithm (EPLL) and its extensions have shown good performances for image denoising. The prior model used by EPLL is usually a Gaussian mixture model (GMM) estimated from a database of image patches. Classical mixture model estimation methods face computational issues as the high dimensionality of the problem requires training on large datasets. In this work, we adapt a compressive statistical learning framework to carry out the GMM estimation. With this method, called sketching, we estimate models from a compressive representation (the sketch) of the training patches. The cost of estimating the prior from the sketch no longer depends on the number of items in the original large database. To accelerate further the estimation, we add another dimension reduction technique (low-rank modeling of the covariance matrices) to the compressing learning framework. To demonstrate the advantages of our method, we test it on real large-scale data. We show that we can produce denoising performances similar to performances obtained with models estimated from the original training database using GMM priors learned from the sketch with improved execution times.

The pansharpening problem amounts to fusing a high-resolution panchromatic image with a low-resolution multispectral image so as to obtain a high-resolution multispectral image. Therefore, the preservation of the spatial resolution of the panchromatic image and the spectral resolution of the multispectral image is of key importance for the pansharpening problem. To cope with it, we propose a new method based on a bidiscriminator in a generative adversarial network (GAN) framework. The first discriminator is optimized to preserve textures of images by taking as input the luminance and the near-infrared band of images, and the second discriminator preserves the color by comparing the chroma components Cb and Cr. Thus, this method allows to train two discriminators, each one with a different and complementary task. Moreover, to enhance these aspects, the proposed method based on bidiscriminator, and called MDSSC-GAN SAM, considers a spatial and a spectral constraint in the loss function of the generator. We show the advantages of this new method on experiments carried out on Pléiades and World View 3 satellite images.

The pansharpening problem consists in fusing a high resolution panchromatic image with a low resolution multispectral image in order to obtain a high resolution multispectral image. In this paper, we adapt a Residual Dense architecture for the generator in a Generative Adversarial Network framework. Indeed, this type of architecture avoids the vanishing gradient problem faced when training a network by re-injecting previous information thanks to dense and residual connections. Moreover, an important point for the pansharpening problem is to preserve the geometry of the image. Hence, we propose to add a regularization term in the loss function of the generator: it preserves the geometry of the target image so that a better solution is obtained. In addition, we propose geometrical measures that illustrate the advantages of this new method.

We propose to detect defects in old movies, as the first step of a larger framework of old movies restoration by inpainting techniques. The specificity of our work is to learn a film restorer's expertise from a pair of sequences, composed of a movie with defects, and the same movie which was semiautomatically restored with the help of a specialized software. In order to detect those defects with minimal human interaction and further reduce the time spent for a restoration, we feed a U-Net with consecutive defective frames as input to detect the unexpected variations of pixel intensity over space and time. Since the output of the network is a mask of defect location, we first have to create the dataset of mask frames on the basis of restored frames from the software used by the film restorer, instead of classical synthetic ground truth, which is not available. These masks are estimated by computing the absolute difference between restored frames and defectuous frames, combined with thresholding and morphological closing. Our network succeeds in automatically detecting real defects with more precision than the manual selection with an all-encompassing shape, including some the expert restorer could have missed for lack of time.

We propose a new algorithm for sparse spike estimation from Fourier measurements. Based on theoretical results on non-convex optimization techniques for off-the-grid sparse spike estimation, we present a projected gradient descent algorithm coupled with a back-projection initialization procedure. Our algorithm permits to estimate the positions of large numbers of Diracs in 2d from random Fourier measurements. We present, along with the algorithm, theoretical qualitative insights explaining the success of our algorithm. This opens a new direction for practical off-the-grid spike estimation with theoretical guarantees in imaging applications.

Nonlinear eigenfunctions, induced by subgradients of one-homogeneous functionals (such as the 1-Laplacian), have shown to be instrumental in segmentation, clustering, and image decomposition. We present a class of flows for finding such eigenfunctions, generalizing a method recently suggested by Nossek and Gilboa. We analyze the flows on grids and graphs in the time-continuous and time-discrete settings. For a specific type of flow within this class, we prove convergence of the numerical iterations procedure and prove existence and uniqueness of the time-continuous case. Several toy examples are provided for illustrating the theoretical results, showing how such flows can be used on images and graphs.

This paper deals with the difficult problem of video colorization. Methods in the literature are generally based on spatio-temporal video blocks or on frame-to-frame color propagation algorithms, each technique having its own advantages and drawbacks. In this paper, we present both a novel automatic frame-to-frame propagation approach and an interactive correction method within a variational framework. The proposed method propagates colors from an initial colorized frame to the whole grayscale video sequence. The automatic propagation results may be visually unsuitable in some cases. To overcome this limitation, a spatio-temporal functional with a user-guided correction is introduced. Two fast primal-dual algorithms are designed to solve the proposed variational models. Numerical results show the efficiency and the potentiality of the proposed approach in comparison with state-of-the-art methods.

In this paper, we aim at super-resolving a low-resolution texture under the assumption that a high-resolution patch of the texture is available. To do so, we propose a variational method that combines two approaches that are texture synthesis and image reconstruction. The resulting objective function holds a nonconvex energy that involves a quadratic distance to the low-resolution image, a histogram-based distance to the high-resolution patch, and a nonlocal regularization that links the missing pixels with the patch pixels. As for the histogram-based measure, we use a sum of Wasserstein distances between the histograms of some linear transformations of the textures. The resulting optimization problem is efficiently solved with a primal-dual proximal method. Experiments show that our method leads to a significant improvement, both visually and numerically, with respect to the state-of-the-art algorithms for solving similar problems.

This paper presents a novel variational image inpainting method to solve the problem of generating, from 3-D lidar measures, a dense depth map coherent with a given color image, tackling visibility issues. When projecting the lidar point cloud onto the image plane, we generally obtain a sparse depth map, due to undersampling. Moreover, lidar and image sensor positions generally differ during acquisition, such that depth values referring to objects that are hidden from the image view point might appear with a naive projection. The proposed algorithm estimates the complete depth map, while simultaneously detecting and excluding those hidden points. It consists in a primal-dual optimization method, where a coupled total variation regularization term is included to match the depth and image gradients and a visibility indicator handles the selection of visible points. Tests with real data prove the effectiveness of the proposed strategy.

This paper proposes a novel model for contrast enhancement of RGB images. The average local contrast measure is increased within a variational framework which preserves the hue of the original image by coupling the channels. The user is enabled to intuitively control the level of the contrast as well as the scale of the enhanced details. Moreover, our model avoids large modifications of the original image histogram and thereby preserves the global illumination of the scene. The minimizer of the proposed functional is computed by a hybrid primal-dual algorithm. Numerical experiments show the reliability of the proposed approach in comparison with state-of-the-art methods.

We consider the problem of recovering a high-resolution image from a pair consisting of a complete low-resolution image and a high-resolution but incomplete one. We refer to this task as the image zoom completion problem. After discussing possible contexts in which this setting may arise, we introduce a nonlocal regularization strategy, giving full details concerning the numerical optimization of the corresponding energy and discussing its benefits and shortcomings. We also derive two total variation-based algorithms and evaluate the performance of the proposed methods on a set of natural and textured images. We compare the results and get with those obtained with two recent state-of-the-art single-image super-resolution algorithms.

This paper provides a new method to colorize gray-scale images. While the computation of the luminance channel is directly performed by a linear transformation, the colorization process is an ill-posed problem that requires some priors. In the literature two classes of approach exist. The first class includes manual methods that need the user to manually add colors on the image to colorize. The second class includes exemplar-based approaches where a color image, with a similar semantic content, is provided as input to the method. These two types of priors have their own advantages and drawbacks. In this paper, a new variational framework for exemplar-based colorization is proposed. A nonlocal approach is used to find relevant color in the source image in order to suggest colors on the gray-scale image. The spatial coherency of the result as well as the final color selection is provided by a nonconvex variational framework based on a total variation. An efficient primal-dual algorithm is provided, and a proof of its convergence is proposed. In this work, we also extend the proposed exemplar-based approach to combine both exemplar-based and manual methods. It provides a single framework that unifies advantages of both approaches. Finally, experiments and comparisons with state-of-the-art methods illustrate the efficiency of our proposal.

We propose a two-step algorithm that automatically estimates the noise level function of stationary noise from a single image, i.e., the noise variance as a function of the image intensity. First, the image is divided into small square regions and a nonparametric test is applied to decide whether each region is homogeneous or not. Based on Kendall's $\tau$ coefficient (a rank-based measure of correlation), this detector has a nondetection rate independent of the unknown distribution of the noise, provided that it is at least spatially uncorrelated. Moreover, we prove, on a toy example, that its overall detection error vanishes with respect to the region size as soon as the signal to noise ratio level is nonzero. Once homogeneous regions are detected, the noise level function is estimated as a second order polynomial minimizing the $\ell^1$ error on the statistics of these regions. Numerical experiments show the efficiency of the proposed approach in estimating the noise level function, with a relative error under 10% obtained on a large data set. We illustrate the interest of the approach for an image denoising application.

The classical super-resolution (SR) setting starts with a set of low-resolution (LR) images related by subpixel shifts and tries to reconstruct a single high-resolution (HR) image. In some cases, partial observations about the HR image are also available. Trying to complete the missing HR data without any reference to LR ones is an inpainting (or completion) problem. In this paper, we consider the problem of recovering a single HR image from a pair consisting of a complete LR and incomplete HR image pair. This setting arises in particular when one wants to fuse image data captured at two different resolutions. We propose an efficient algorithm that allows to take advantage of both image data by first learning nonlocal interactions from an interpolated version of the LR image using patches. Those interactions are then used by a convex energy function whose minimization yields a super-resolved complete image.

This paper deals with the problem of image colorization. A model including total variation regularization is proposed. Our approach colorizes directly the three RGB channels, while most existing methods were only focusing on the two chrominance channels. By using the three channels, our approach is able to better preserve color consistency. Our model is non convex, but we propose an efficient primal-dual like algorithm to compute a local minimizer. Numerical examples illustrate the good behavior of our algorithm with respect to state-of-the-art methods.

We derive a denoising method based on an adaptive regularization of the non-local means. The NL-means reduce noise by using the redundancy in natural images. They compute a weighted average of pixels whose surroundings are close. This method performs well but it suffers from residual noise on singular structures. We use the weights computed in the NL-means as a measure of performance of the denoising process. These weights balance the data-fidelity term in an adapted ROF model, in order to locally perform adaptive TV regularization. Besides, this model can be adapted to different noise statistics and a fast resolution can be computed in the general case of the exponential family. We adapt this model to video denoising by using spatio-temporal patches. Compared to spatial patches, they offer better temporal stability, while the adaptive TV regularization corrects the residual noise observed around moving structures.

This article introduces a generalization of the discrete optimal transport, with applications to color image manipulations. This new formulation includes a relaxation of the mass conservation constraint and a regularization term. These two features are crucial for image processing tasks where one must take into account families of multimodal histograms with large mass variation across modes. The corresponding relaxed and regularized transportation problem is the solution of a convex optimization problem. Depending on the regularization used, this minimization can be solved using standard linear programming methods or first order proximal splitting schemes. The resulting transportation plan can be used as a color transfer map, which is robust to mass variation across image color palettes. Furthermore, the regularization of the transport plan helps remove colorization artifacts due to noise amplification. We also extend this framework to compute the barycenter of distributions. The barycenter is the solution of an optimization problem, which is separately convex with respect to the barycenter and the transportation plans, but not jointly convex. A block coordinate descent scheme converges to a stationary point of the energy. We show that the resulting algorithm can be used for color normalization across several images. The relaxed and regularized barycenter defines a common color palette for those images. Applying color transfer toward this average palette performs a color normalization of the input images.

This paper addresses the problem of modeling textures with Gaussian processes, focusing on color stationary textures that can be either static or dynamic. We detail two classes of Gaussian processes parameterized by a small number of compactly supported linear filters, the so-called textons. The first class extends the spot noise texture model to the dynamical setting, where the space-time texton is estimated to fit a translation-invariant covariance from an input exemplar. The second class is a specialization of the autoregressive dynamic texture method to the setting of space- and time-stationary textures. This enables one to parameterize the covariance with only a few spatial textons. The simplicity of these models allows us to tackle a more complex problem, texture mixing, which, in our case, amounts to interpolating between Gaussian models. We use optimal transport to derive geodesic paths and barycenters between the models learned from an input data set. This enables the user to navigate inside the set of texture models and perform texture synthesis from each new interpolated model. Numerical results on a library of exemplars show the ability of our method to generate arbitrary interpolations among unstructured natural textures. Moreover, experiments on a database of stationary textures show that the methods, despite their simplicity, provide state-of-the-art results on stationary dynamical texture synthesis and mixing.

Image denoising is a central problem in image processing and it is often a necessary step prior to higher level analysis such as segmentation, reconstruction, or super-resolution. The nonlocal means (NL-means) perform denoising by exploiting the natural redundancy of patterns inside an image; they perform a weighted average of pixels whose neighborhoods (patches) are close to each other. This reduces significantly the noise while preserving most of the image content. While it performs well on flat areas and textures, it suffers from two opposite drawbacks: it might over-smooth low-contrasted areas or leave a residual noise around edges and singular structures. Denoising can also be performed by total variation minimization-the Rudin, Osher and Fatemi model-which leads to restore regular images, but it is prone to over-smooth textures, staircasing effects, and contrast losses. We introduce in this paper a variational approach that corrects the over-smoothing and reduces the residual noise of the NL-means by adaptively regularizing nonlocal methods with the total variation. The proposed regularized NL-means algorithm combines these methods and reduces both of their respective defaults by minimizing an adaptive total variation with a nonlocal data fidelity term. Besides, this model adapts to different noise statistics and a fast solution can be obtained in the general case of the exponential family. We develop this model for image denoising and we adapt it to video denoising with 3D patches.

This paper is concerned with the relaxation of nonconvex functionals used in image processing. We review most of the recently introduced relaxation methods, and we propose a new convex one based on a probabilistic approach, which has the advantages of being intuitive, flexible, and involving an algorithm without inner loops. We investigate in detail the connections between the solutions of the relaxed functionals with minimizers of the original one. Such connection is demonstrated only for a nonconvex relaxation which turns out to be quite robust to initialization. As a case of study, we illustrate our theoretical analysis with numerical experiments, namely, for the optical flow problem.

This paper addresses the problem of modeling stationary color dynamic textures with Gaussian processes. We detail two particular classes of such processes that are parameterized by a small number of compactly supported linear filters, so-called dynamical textons (dynTextons). The first class extends previous works on the spot noise texture model to the dynamical setting. It directly estimates the dynTexton to fit a translation-invariant covariance from the exemplar. The second class is a specialization of the auto-regressive (AR) dynamic texture method to the setting of space and time stationary textures. This allows one to parameterize the process covariance using only a few linear filters. Numerical experiments on a database of stationary textures shows that the methods, despite their extreme simplicity, provide state of the art results to synthesize space stationary dynamical texture.

This paper deals with the problem of segmenting an image with active contours. We explain how recent convexification methods allow now to use active contours without level sets with simple and efficient first order schemes. We recall different algorithms proposed in the literature, and we propose a new variant. Numerical experiments in 2D and 3D confirm the interest of the approach.

Cone Beam Computerized Tomography (CBCT) and Positron Emission Tomography (PET) Scans are medical imaging devices that require solving ill-posed inverse problems. The models considered come directly from the physics of the acquisition devices, and take into account the specificity of the (Poisson) noise. We propose various fast numerical schemes to compute the solution. In particular, we show that a new algorithm recently introduced by A. Chambolle and T. Pock is well suited in the PET case when considering non differentiable regularizations such as total variation or wavelet ℓ1-regularization. Numerical experiments indicate that the proposed algorithms compare favorably with respect to well-established methods in tomography.

This paper deals with the parameter choice for the nonlocal means (NLM) algorithm. After basic computations on toy models highlighting the bias of the NLM, we study the bias-variance trade-off of this filter so as to highlight the need of a local choice of the parameters. Relying on Stein's unbiased risk estimate, we then propose an efficient algorithm to locally set these parameters, and we compare this method with the NLM with optimal global parameter.

This article presents a new adaptive framework for locally parallel texture modeling. Oscillating patterns are modeled with functionals that constrain the local Fourier decomposition of the texture. We first introduce a texture functional which is a weighted Hilbert norm. The weights on the local Fourier atoms are optimized to match the local orientation and frequency of the texture. This adaptive model is used to solve image processing inverse problems, such as image decomposition and inpainting. The local orientation and frequency of the texture component are adaptively estimated during the minimization process. To improve inpainting performances over large missing regions, we introduce a highly nonconvex generalization of our texture model. This new model constrains the amplitude of the texture and allows one to impose an arbitrary oscillation profile. Numerical results illustrate the effectiveness of the method.

Space agencies are rapidly building up massive image databases. A particularity of these databases is that they are made of images with different, but known, resolutions. In this paper, we introduce a new scheme allowing us to compare and index images with different resolutions. This scheme relies on a simplified acquisition model of satellite images and uses continuous wavelet decompositions. We establish a correspondence between scales which permits us to compare wavelet decompositions of images having different resolutions. We validate the approach through several matching and classification experiments, and we show that taking the acquisition process into account yields better results than just using scaling properties of wavelet features.

We study the problem of finding the characteristic scale of a given satellite image. This feature is defined so that it does not depend on the spatial resolution of the image. This is a different problem than achieving scale invariance, as often studied in the literature. Our approach is based on the use of a linear scale space and the total variation (TV). The critical scale is defined as the one at which the normalized TV reaches its maximum. It is shown experimentally, both on synthetic and real data, that the computed characteristic scale is resolution independent.

We study the problem of finding the characteristic scale of a given satellite image. We want to define this feature so that it does not depend on the spatial resolution of the image. Our approach is based on the use of a linear scale space and the total variation. The critical scale is defined as the one at which the normalized total variation is maximum.

In this paper, we propose a new scheme to extrapolate wavelet features with respect to the resolution. By explicitly taking into account the acquisition process of satellite images, we compute how wavelet features behave when the resolution changes. This approach is validated by classifying satellite images with different resolutions.

A supervised classification model based on a variational approach is presented. This model is specifically devoted to textured images. We want to get a partition of an image, composed of texture regions separated by regular interfaces. Each kind of texture defines a class. We use a wavelet packet transform to analyze the textures, characterized by their energy distribution in each sub-band. In order to have an image segmentation according to the classes, we model the regions and their interfaces by level set functions. We define a functional on these level sets whose minimizers define the optimal classification according to textures. A system of coupled PDEs is deduced from the functional. By solving this system, each region evolves according to its wavelet coefficients and interacts with the neighbour region in order to obtain a partition with regular contours. Experiments are shown on synthetic and real images.

We present a supervised classification model based on a variational approach. This model is specifically devoted to textured images. We want to get a partition of an image, composed of texture regions separated by regular interfaces. Each kind of texture defines a class. We use a wavelet packet transform to analyze the textures, characterized by their energy distribution in each sub-band. In order to have an image segmentation according to the classes, we model the regions and their interfaces by level set functions. We define a functional on these level sets whose minimizers define the optimal classification according to texture. A system of coupled PDEs is deduced from the functional. By solving this system, each region evolves according to its wavelet coefficients and interacts with the neighbor regions in order to obtain a partition with regular contours. Experiments are shown on synthetic and real images.