This paper presents LatentPatch, a new method for generating realistic images from a small dataset of only a few images. We use a lightweight model with only a few thousand parameters. Unlike traditional few-shot generation methods that finetune pre-trained large-scale generative models, our approach is computed directly on the latent distribution by sequential feature matching, and is explainable by design. Avoiding large models based on transformers, recursive networks, or self-attention, which are not suitable for small datasets, our method is inspired by non-parametric texture synthesis and style transfer models, and ensures that generated image features are sampled from the source distribution. We extend previous single-image models to work with a few images and demonstrate that our method can generate realistic images, as well as enable conditional sampling and image editing. We conduct experiments on face datasets and show that our simplistic model is effective and versatile.

With the emergence of deep perceptual image features, style transfer has become a popular application that repaints a picture while preserving the geometric patterns and textures from a sample image. Our work is devoted to the combination of perceptual features from multiple style images, taken at different scales, e.g. to mix large-scale structures of a style image with fine-scale textures. Surprisingly, this turns out to be difficult, as most deep neural representations are learned to be robust to scale modifications, so that large structures tend to be tangled with smaller scales. Here a multi-scale convolutional architecture is proposed for bi-scale style transfer. Our solution is based on a modular auto-encoder composed of two lightweight modules that are trained independently to transfer style at specific scales, with control over styles and colors.

Since a few years and the advent of convolutional neural networks, algorithms for artistic style transfer between images have developed considerably. However, these methods require a relatively long training phase in order to succeed. This is why non-learning image processing approaches recently strove to propose patch-based algorithms able to aesthetically compete with neural methods. This paper goes one step further in this direction by introducing a new patch-based method for style transfer, using a constrained multi-scale version of the fast approximate nearest-neighbor algorithm PatchMatch, enforcing uniform sampling of style featurepatch. Our method also aims to mix the patch-based and neural paradigms by enabling the embedding of image patches in the feature space of the VGG-16 network.

In this paper, a new automatic method for the illumination of flat-colored drawings is proposed. First, we reconstruct a 3D augmentation of a 2D silhouette from the analysis of its skeleton. Then, we apply the Phong lighting model that relies on the estimated normal map to generate an illuminated drawing. This method compares favorably to recent state-of-the-art methods, e.g. those using convolutional neural networks.

Image Super Resolution (SR) has come a long way since the early age of image processing. Deep learning methods nowadays give outstanding results, yet very few are actually used in digital illustration and photo retouching software due to large memory storage and GPU computational requirements, but also due to the actual lack of control provided to the user over the final result. This paper introduces a two-step framework for stylized SR using a multi-scale network built with independent parallel branches. The approach aims at: i. designing a shallow network based on image processing techniques making it usable on light hardware architecture (low memory cost, no GPU); ii. providing a versatile, controllable and customizable network to stylize SR results in a plug-and-play manner. We show that the proposed method offers significant advantages over state-of-the-art reference-based approaches regarding these aspects.

Three-dimensional (3D) CLUTs (color lookup tables) are popular digital models used in artistic image and video processing for color grading, simulation of analog films, and more generally the description and application of generic nonparametric color transformations. The relatively large size of these models leads to high data storage requirements when trying to distribute them on a large scale (e.g., several hundred at a time). In this article, an effective technique based on a multiscale anisotropic diffusion scheme is proposed, for the lossy compression of generic CLUTs regularly sampled on a 3D grid. Our method exhibits high average compression rates, while ensuring visually indistinguishable differences with the original ( uncompressed) CLUTs. In a second step, a variation of our algorithm for exemplar-based generation of CLUTs is developed in order to create a complete CLUT from a single pair of before/after images that accounts for the color transformation.

We propose to tackle the problem of RGB-D image disocclusion inpainting when synthesizing new views of a scene by changing its viewpoint. Indeed, such a process creates holes both in depth and color images. First, we propose a novel algorithm to perform depth-map disocclusion inpainting. Our intuitive approach works particularly well for recovering the lost structures of the objects and to inpaint the depth-map in a geometrically plausible manner. Then, we propose a depth-guided patch-based inpainting method to fill-in the color image. Depth information coming from the reconstructed depth-map is added to each key step of the classical patch-based algorithm from Criminisi et al. in an intuitive manner. Relevant comparisons to the state-of-the-art inpainting methods for the disocclusion inpainting of both depth and color images are provided and illustrate the effectiveness of our proposed algorithms.

In this paper we propose an approach to inpaint holes in depth maps that appear when synthesizing virtual views from a RGB-D scenes. Based on a superpixel oversegmentation of both the original and synthesized views, the proposed approach efficiently deals with many occlusion situations where most of previous approaches fail. The use of superpixels makes the algorithm more robust to inaccurate depth maps, while giving an efficient way to model the image. Extensive comparisons to relevant state-of-the-art methods show that our approach outperforms qualitatively and quantitavely these existing approaches.

This paper proposes a technical review of exemplar-based inpainting approaches with a particular focus on greedy methods. Several comparative and illustrative experiments are provided to deeply explore and enlighten these methods, and to have a better understanding on the state-of-the-art improvements of these approaches. From this analysis, three improvements over Criminisi et al. algorithm are then presented and detailed: 1) a tensor-based data term for a better selection of pixel candidates to fill in; 2) a fast patch lookup strategy to ensure a better global coherence of the reconstruction; and 3) a novel fast anisotropic spatial blending algorithm that reduces typical block artifacts using tensor models. Relevant comparisons with the state-of-the-art inpainting methods are provided that exhibit the effectiveness of our contributions.

In this paper, we propose two major improvements to the exemplar-based image inpainting algorithm, initially formulated by Criminisi et al. [1]. First, we introduce a structure-tensor-based data-term for a better selection of pixel candidates to fill in based on priority. Then, we propose a new lookup heuristic in order to locate the best source patches to copy/paste to these targeted points. These two contributions clearly make the inpainting algorithm reconstruct more geometrically coherent images, as well as speed up the process drastically. We illustrate the great performances of our approach compared to existing state-of-the-art methods.

We describe a simple but capable algorithm to generate sketches from color images. The method is based on the drawing of dark strokes on a white background, directed by an image-dependent tensor-valued geometry. It is able to produce various sketching styles with different kind of hatching and contours. Combining these grayscale sketches with the colors of the input images allows to produce a wide variety of image stylization renderings.

We design a family of non-local image smoothing algorithms which approximate the application of diffusion PDE's on a specific Euclidean space of image patches. We first map a noisy image onto this high-dimensional space and estimate its geometric structure thanks to a straightforward extension of the structure tensor field. The tensors spectral elements allows us to design an oriented high-dimensional smoothing process by the means of anisotropic regularization PDE's which have both local and non-local properties and whose solutions are estimated by locally oriented high-dimensional convolutions. We show that the Bilateral Filtering and Non-Local Means methods are the isotropic cases of our denoising framework.

We present a robust method to retrieve neuronal fibers in human brain white matter from high-angular resolution MRI (HARDI datasets). Contrary to classical fiber-tracking techniques done on the traditional 2nd-order tensor model (DTI) which may lead to truncated or biased estimated diffusion directions in case of fiber crossing configurations, we propose here a more complex approach based on a variational estimation of orientation diffusion functions (ODF) modeled with spherical harmonics. This kind of model can correctly retrieve multiple fiber directions corresponding to underlying intra-voxel fibers populations. Our technique is able to consider the Rician noise model of the MRI acquisition in order to better estimate the white matter fiber tracks. Results on both synthetic and real human brain white matter HARDI datasets illustrate the effectiveness of the proposed approach.

We address the problem of temporal tracking fine pointlike and filamentary structures exhibiting smooth motions in image sequences. By taking these specific restrictions into account, we put forward an original tracking method based on the search of integral lines in time-space structure tensor fields. The method is simple and very efficient regarding computation time and tracking precision, which allows a sub-pixel accuracy in both spatial and temporal domains. We suggest a numerical implementation of the algorithm which is based on a modified constrained Runge-Kutta scheme. Its performance and potential applicability are illustrated through two real cases : the tracking of internal linear features in composite materials observed with X-rays and the tracking of granular-shaped objects moving in a gas flow.

This paper introduces a complete pipeline of image processing methods in order to analyze and track the internal structures of a composite material. As a first step, input Xray images are denoised, enhanced, separated into different morphological components, and geometrically filtered in order to isolate the interesting fiber inside the composite material. This requires the design of specific algorithms preserving very thin image details while being able to remove undesired image regions that may be sometimes large. For this purpose, we use state-of the-art techniques based on recent achievements in diffusion PDE’s and modern harmonic analysis tools. Then, the shapes of the remaining fibered composite are individually analyzed by the use of a tensor-based tracking algorithm. We illustrate how this set of techniques allows the dynamic analysis of the composite structure when submitted to external mechanical loads.

In this paper, a general multi-valued image regularization method based on LIC's (line integral convolutions [B. Cabral, 1993]} is proposed. From the investigation of recent approaches based on multi-valued diffusion PDE's, we show how a regularization process is naturally decomposed, first as the estimation of its underlying smoothing geometry, and then, as the application of a locally and spatially oriented smoothing. Performing this last part using LIC's significatively improves the overall regularization process both in visual quality and processing time. We illustrate three different applications of our general regularization framework: color image denoising, inpainting and magnification.

In this paper, we focus on techniques for vector-valued image regularization, based on variational methods and PDEs. Starting from the study of PDE-based formalisms previously proposed in the literature for the regularization of scalar and vector-valued data, we propose a unifying expression that gathers the majority of these previous frameworks into a single generic anisotropic diffusion equation. On one hand, the resulting expression provides a simple interpretation of the regularization process in terms of local filtering with spatially adaptive Gaussian kernels. On the other hand, it naturally disassembles any regularization scheme into the smoothing process itself and the underlying geometry that drives the smoothing. Thus, we can easily specialize our generic expression into different regularization PDEs that fulfill desired smoothing behaviors, depending on the considered application: image restoration, inpainting, magnification, flow visualization, etc. Specific numerical schemes are also proposed, allowing us to implement our regularization framework with accuracy by taking the local filtering properties of the proposed equations into account. Finally, we illustrate the wide range of applications handled by our selected anisotropic diffusion equations with application results on color images.

We address the problem of vector-valued image regularization with variational methods and PDEs. From the study of existing formalisms, we propose a unifying framework based on a very local interpretation of the regularization processes. The resulting equations are then specialized into new regularization PDEs and corresponding numerical schemes that respect the local geometry of vector-valued images. They are finally applied on a wide variety of image processing problems, including color image restoration, in-painting, magnification and flow visualization.

We address three crucial issues encountered in DT-MRI (diffusion tensor magnetic resonance imaging): diffusion tensor estimation, regularization and fiber bundle visualization. We first review related algorithms existing in the literature and propose then alternative variational formalisms that lead to new and improved schemes, thanks to the preservation of important tensor constraints (positivity, symmetry). We illustrate how our complete DT-MRI processing pipeline can be successfully used to construct and draw fiber bundles in the white matter of the brain, from a set of noisy raw MRl images.

The paper deals with the problem of regularizing noisy fields of diffusion tensors, considered as symmetric and semi-positive definite n /spl times/ n matrices (such as for instance 2D structure tensors or DT-MRI medical images). We first propose a simple anisotropic, PDE-based scheme that acts directly on the matrix coefficients and preserves the semi-positive constraint thanks to a specific reprojection step. The limitations of this algorithm lead us to introduce a more effective approach based on constrained spectral regularizations acting on the tensor orientations (eigenvectors) and diffusivities (eigenvalues), while explicitly taking the tensor constraints into account. The regularization of the orientation part uses orthogonal matrix diffusion PDE's and local vector alignment procedures. For the interesting 3D case, a special implementation scheme designed to numerically fit the tensor constraints is also proposed. Experimental results on synthetic and real DT-MRI data sets finally illustrates the proposed tensor regularization framework.