This paper presents LatentPatch, a new method for generating realistic images from a small dataset of only a few images. We use a lightweight model with only a few thousand parameters. Unlike traditional few-shot generation methods that finetune pre-trained large-scale generative models, our approach is computed directly on the latent distribution by sequential feature matching, and is explainable by design. Avoiding large models based on transformers, recursive networks, or self-attention, which are not suitable for small datasets, our method is inspired by non-parametric texture synthesis and style transfer models, and ensures that generated image features are sampled from the source distribution. We extend previous single-image models to work with a few images and demonstrate that our method can generate realistic images, as well as enable conditional sampling and image editing. We conduct experiments on face datasets and show that our simplistic model is effective and versatile.

The recent advent of powerful generative models has triggered the renewed development of quantitative measures to assess the proximity of two probability distributions. As the scalar Frechet Inception Distance remains popular, several methods have explored computing entire curves, which reveal the trade-off between the fidelity and variability of the first distribution with respect to the second one. Several of such variants have been proposed independently and while intuitively similar, their relationship has not yet been made explicit. In an effort to make the emerging picture of generative evaluation more clear, we propose a unification of four curves known respectively as: the Precision-Recall (PR) curve, the Lorenz curve, the Receiver Operating Characteristic (ROC) curve and a special case of Rényi divergence frontiers. In addition, we discuss possible links between PR / Lorenz curves with the derivation of domain adaptation bounds.

With the emergence of deep perceptual image features, style transfer has become a popular application that repaints a picture while preserving the geometric patterns and textures from a sample image. Our work is devoted to the combination of perceptual features from multiple style images, taken at different scales, e.g. to mix large-scale structures of a style image with fine-scale textures. Surprisingly, this turns out to be difficult, as most deep neural representations are learned to be robust to scale modifications, so that large structures tend to be tangled with smaller scales. Here a multi-scale convolutional architecture is proposed for bi-scale style transfer. Our solution is based on a modular auto-encoder composed of two lightweight modules that are trained independently to transfer style at specific scales, with control over styles and colors.

Since a few years and the advent of convolutional neural networks, algorithms for artistic style transfer between images have developed considerably. However, these methods require a relatively long training phase in order to succeed. This is why non-learning image processing approaches recently strove to propose patch-based algorithms able to aesthetically compete with neural methods. This paper goes one step further in this direction by introducing a new patch-based method for style transfer, using a constrained multi-scale version of the fast approximate nearest-neighbor algorithm PatchMatch, enforcing uniform sampling of style featurepatch. Our method also aims to mix the patch-based and neural paradigms by enabling the embedding of image patches in the feature space of the VGG-16 network.

In this work, we propose a new parameter efficient sharing method for the training of GAN generators. While there has been recent progress in transfer learning for generative models with limited data, they are either limited to domains close to the original one, or adapt a large part of the parameters. This is somewhat redundant, as the goal of transfer learning should be to reuse old features. In this way, we propose width wise parameter sharing, which can learn a new domain with ten times fewer trainable parameters without a significant drop in quality. Previous approaches are less flexible than our method and also fail to preserve image quality for challenging transfers. Finally, as our goal is ultimately parameter reuse, we show that our method performs well in the multi-domain setting, wherein several domains are learned simultaneously with higher visual quality than the state of the art StarGAN-V2.

Image Super Resolution (SR) has come a long way since the early age of image processing. Deep learning methods nowadays give outstanding results, yet very few are actually used in digital illustration and photo retouching software due to large memory storage and GPU computational requirements, but also due to the actual lack of control provided to the user over the final result. This paper introduces a two-step framework for stylized SR using a multi-scale network built with independent parallel branches. The approach aims at: i. designing a shallow network based on image processing techniques making it usable on light hardware architecture (low memory cost, no GPU); ii. providing a versatile, controllable and customizable network to stylize SR results in a plug-and-play manner. We show that the proposed method offers significant advantages over state-of-the-art reference-based approaches regarding these aspects.

With the widespread application of deep networks in industry, membership inference attacks, i.e. the ability to discern training data from a model, become more and more problematic for data privacy. Recent work suggests that generative networks may be robust against membership attacks. In this work, we build on this observation, offering a general-purpose solution to the membership privacy problem. As the primary contribution, we demonstrate how to construct surrogate datasets, using images from GAN generators, labelled with a classifier trained on the private dataset. Next, we show this surrogate data can further be used for a variety of downstream tasks (here classification and regression), while being resistant to membership attacks. We study a variety of different GANs proposed in the literature, concluding that higher quality GANs result in better surrogate data with respect to the task at hand.

Abstract This paper describes a novel approach for on demand volumetric texture synthesis based on a deep learning framework that allows for the generation of high‐quality three‐dimensional (3D) data at interactive rates. Based on a few example images of textures, a generative network is trained to synthesize coherent portions of solid textures of arbitrary sizes that reproduce the visual characteristics of the examples along some directions. To cope with memory limitations and computation complexity that are inherent to both high resolution and 3D processing on the GPU, only 2D textures referred to as ‘slices’ are generated during the training stage. These synthetic textures are compared to exemplar images via a perceptual loss function based on a pre‐trained deep network. The proposed network is very light (less than 100k parameters), therefore it only requires sustainable training (i.e. few hours) and is capable of very fast generation (around a second for 2563 voxels) on a single GPU. Integrated with a spatially seeded pseudo‐random number generator (PRNG) the proposed generator network directly returns a color value given a set of 3D coordinates. The synthesized volumes have good visual results that are at least equivalent to the state‐of‐the‐art patch‐based approaches. They are naturally seamlessly tileable and can be fully generated in parallel.

State of the art deep generative networks have achieved such realism that they can be suspected of memorizing training images. It is why it is not uncommon to include visualizations of training set nearest neighbors, to suggest generated images are not simply memorized. We argue this is not sufficient and motivates studying overfitting of deep generators with more scrutiny. We address this question by i) showing how simple losses are highly effective at reconstructing images for deep generators ii) analyzing the statistics of reconstruction errors for training versus validation images. Using this methodology, we show that pure GAN models appear to generalize well, in contrast with those using hybrid adversarial losses, which are amongst the most widely applied generative methods. We also show that standard GAN evaluation metrics fail to capture memorization for some deep generators. Finally, we note the ramifications of memorization on data privacy. Considering the already widespread application of generative networks, we provide a step in the right direction towards the important yet incomplete picture of generative overfitting.

In this article we revisit the definition of Precision-Recall (PR) curves for generative models proposed by (Sajjadi et al., 2018). Rather than providing a scalar for generative quality, PR curves distinguish mode-collapse (poor recall) and bad quality (poor precision). We first generalize their formulation to arbitrary measures hence removing any restriction to finite support. We also expose a bridge between PR curves and type I and type II error (a.k.a. false detection and rejection) rates of likelihood ratio classifiers on the task of discriminating between samples of the two distributions. Building upon this new perspective, we propose a novel algorithm to approximate precision-recall curves, that shares some interesting methodological properties with the hypothesis testing technique from (Lopez-Paz & Oquab, 2017). We demonstrate the interest of the proposed formulation over the original approach on controlled multi-modal datasets.

This paper is devoted to the study of the discrete formulations of the weighted Total Variation (TV) based on upwind schemes that have been proposed for imaging problems in a local setting in [1] and in a non-local setting for graphs and point-clouds in [2]. We focus on two new symmetric formulations based on the l2and l∞norms respectively and propose a dedicated optimization algorithm to solve convex problems based on such TV penalties. We demonstrate the theoretical and practical interest of such formulations for image processing tasks.

Exemplar-based texture synthesis consists in producing new synthetic images which have the same perceptual characteristics as a given texture sample while exhibiting sufficient innovation (to avoid verbatim copy). In this paper, we propose to address this problem with a model obtained as local transformations of Gaussian random fields. The local transformations operate on $3 \times 3$ patches and are designed to solve a semi-discrete optimal transport problem in order to reimpose the patch distribution of the exemplar texture. The semi-discrete optimal transport problem is solved with a stochastic gradient algorithm, whose convergence speed is evaluated on several practical transport cases. After studying the properties of such transformed Gaussian random fields, we propose a multiscale extension of the model which aims at preserving the patch distribution of the exemplar texture at multiple scales. Experiments demonstrate that this multiscale model is able to synthesize structured textures while keeping several mathematical guarantees, and with low requirements in synthesis time and memory storage. In particular, a single patch optimal transport map is shown to be better than iterated nearest neighbor assignments in terms of statistical guarantees. Besides, once the model is estimated, the resulting synthesis algorithm is fast and highly parallel since it amounts to performing weighted nearest neighbor patch assignments at each scale.

This paper studies the problem of color transfer between images using optimal transport techniques. While being a generic framework to handle statistics properly, it is also known to be sensitive to noise and outliers, and is not suitable for direct application to images without additional postprocessing regularization to remove artifacts. To tackle these issues, we propose to directly deal with the regularity of the transport map and the spatial consistency of the reconstruction. Our approach is based on the relaxed and regularized discrete optimal transport method of [1]. We extend this work by (i) modeling the spatial distribution of colors within the image domain and (ii) tuning automatically the relaxation parameters. Experiments on real images demonstrate the capacity of our model to adapt itself to the considered data.

In this paper, we propose a novel and rigorous framework for region-based active contours that combines the Wasserstein distance between statistical distributions in arbitrary dimension and shape derivative tools. To speed-up the computation and be able to handle high-dimensional features and large-scale data, we introduce an approximation of the differential of the Wasserstein distance between histograms. The framework is flexible enough to allow either minimization of the Wasserstein distance to prior distributions, or maximization of the distance between the distributions of the regions to be segmented (i.e. region competition). Numerical results reported demonstrate the advantages of the proposed optimal transport distance with respect to point-wise metrics.

This paper introduces a novel and generic framework embedding statistical constraints for variational problems. We resort to the theory of Monge-Kantorovich optimal mass transport to define penalty terms depending on statistics from images. To cope with the computation time issue of the corresponding Wasserstein distances involved in this approach, we propose an approximate variational formulation for statistics represented as point clouds. We illustrate this framework on the problem of regularized color specification. This is achieved by combining the proposed approximate Wasserstein constraint on color statistics with a generic geometric-based regularization term in a unified variational minimization problem. We believe that this methodology may lead to some other interesting applications in image processing, such as medical imaging modification, texture synthesis, etc.

This work is concerned with the modification of the gray level or color distribution of digital images. A common drawback of classical methods aiming at such modifications is the revealing of artefacts or the attenuation of details and textures. In this work, we propose a generic filtering method enabling, given the original image and the radiometrically corrected one, to suppress artefacts while preserving details. The approach relies on the key observation that artefacts correspond to spatial irregularity of the so-called transportation map, defined as the difference between the original and the corrected image. The proposed method draws on the nonlocal Yaroslavsky filter to regularize the transportation map. The efficiency of the method is shown on various radiometric modifications: contrast equalization, midway histogram, color enhancement, and color transfer. A comparison with related approaches is also provided.

In this paper, we take interest in the process of assigning a given color distribution to an image. Two examples of such image modifications are histogram equalization (or specification) and color transfer, in which the color palette of a style image is assigned to a source image. Classical methods for gray level specification, as well as more recent methods for color transfer, can be defined as optimal transportation problems. The corresponding image modifications are known to produce visually unpleasing effects such as the removal of details and texture, as well as the enhancement of noise or compression patterns. In this paper, a new method is proposed for the suppression of these artifacts. The method relies on a non local regularization of the transportation map, defined as the difference between the original image and the modified one. The interest of using this method is demonstrated on the aforementioned applications: contrast adjustment and color transfer.

This paper focuses on the matching of local features between images. Given a set of query descriptors and a database of candidate descriptors, the goal is to decide which ones should be matched. This is a crucial issue, since the matching procedure is often a preliminary step for object detection or image matching. In practice, this matching step is often reduced to a specific threshold on the Euclidean distance to the nearest neighbor. Our first contribution is a robust distance between descriptors, relying on the adaptation of the Earth Mover's Distance to circular histograms. It is shown that this distance outperforms classical distances for comparing SIFT-like descriptors, while its time complexity remains reasonable. Our second and main contribution is a statistical framework for the matching procedure, which yields validation thresholds automatically adapted to the complexity of each query descriptor and to the diversity and size of the database. The method makes it possible to detect multiple occurrences, as well as to deal with situations where the target is not present. Its performances are tested through various experiments on a large image database.

In this paper, the matching of SIFT-like features [5] between images is studied. The goal is to decide which matches between descriptors of two datasets should be selected. This matching procedure is often a preliminary step towards some computer vision applications, such as object detection and image registration for instance. The distances between the query descriptors and the database candidates being computed, the classical approach is to select for each query its nearest neighbor, depending on a global threshold on dissimilarity measure. In this contribution, an a contrario framework for the matching procedure is introduced, based on a threshold on a probability of false detections. This approach yields dissimilarity thresholds automatically adapted to each query descriptor and to the diversity and size of the database. We show on various experiments on a large image database, the ability of such a method to decide whether a query and its candidates should be matched.

Many computer vision algorithms make use of local features, and rely on a systematic comparison of these features. The chosen dissimilarity measure is of crucial importance for the overall performances of these algorithms and has to be both robust and computationally efficient. Some of the most popular local features (like SIFT [4] descriptors) are based on one-dimensional circular histograms. In this contribution, we present an adaptation of the Earth moverpsilas distance to one-dimensional circular histograms. This distance, that we call CEMD, is used to compare SIFT-like descriptors. Experiments over a large database of 3 million descriptors show that CEMD outperforms classical bin-to-bin distances, while having reasonable time complexity.