Existing image inpainting methods have shown impressive completion results for low-resolution images. However, most of these algorithms fail at high resolutions and require powerful hardware, limiting their deployment on edge devices. Motivated by this, we propose the first baseline for REal-Time High-resolution image INpainting on Edge Devices (RETHINED) that is able to inpaint at ultra-high-resolution and can run in real-time (< 30ms) in a wide variety of mobile devices. A simple, yet effective novel method formed by a lightweight Convolutional Neural Network (CNN) to recover structure, followed by a resolution-agnostic patch replacement mechanism to provide detailed texture. Specially our pipeline leverages the structural capacity of CNN and the high-level detail of patch-based methods, which is a key component for high-resolution image inpainting. To demonstrate the real application of our method, we conduct an extensive analysis on various mobile-friendly devices and demonstrate similar inpainting performance while being 100 × faster than existing state-of-the-art methods. Furthemore, we realease DF8K-Inpainting, the first free-form mask UHD inpainting dataset. Code and dataset here.

In this paper, we introduce a new variational model for color image restoration, called DIP-VBTV, which combines two priors: a deep image prior (DIP), which assumes that the restored image can be generated through a neural network, and a vector bundle total variation (VBTV), which generalizes the vectorial total variation (VTV) on vector bundles. VBTV is determined by a geometric triplet: a Riemannian metric on the base manifold, a covariant derivative, and a metric on the vector bundle. Whereas the VTV prior encourages the restored images to be piecewise constant, the VBTV prior encourages them to be piecewise parallel with respect to a covariant derivative. For well-chosen geometric triplets, we show that the minimization of VBTV encourages the solutions of the restoration model to share some visual content with the clean image. Then, we show in experiments that DIP-VBTV benefits from this property by outperforming DIP-VTV and state-of-the-art unsupervised methods. It demonstrates the relevance of combining DIP and VBTV priors.

Although orientation has proven to be a key skill of soccer players in order to succeed in a broad spectrum of plays, body orientation is a yet-little-explored area in sports analytics' research. Despite being an inherently ambiguous concept, player orientation can be defined as the projection (2D) of the normal vector placed in the center of the upper-torso of players (3D). This research presents a novel technique to obtain player orientation from monocular video recordings by mapping pose parts (shoulders and hips) in a 2D field by combining OpenPose with a super-resolution network, and merging the obtained estimation with contextual information (ball position). Results have been validated with players-held EPTS devices, obtaining a median error of 27 degrees/player. Moreover, three novel types of orientation maps are proposed in order to make raw orientation data easy to visualize and understand, thus allowing further analysis at team- or player-level.

The colorization of grayscale images is an ill-posed problem, with multiple correct solutions. In this paper, we propose an adversarial learning colorization approach coupled with semantic information. A generative network is used to infer the chromaticity of a given grayscale image conditioned to semantic clues. This network is framed in an adversarial model that learns to colorize by incorporating perceptual and semantic understanding of color and class distributions. The model is trained via a fully self-supervised strategy. Qualitative and quantitative results show the capacity of the proposed method to colorize images in a realistic way achieving state-of-the-art results.

This paper proposes a novel patch-based variational segmentation method that considers adaptive patches to characterize, in an affine invariant way, the local structure of each homogeneous texture region of the image and thus being capable of grouping the same kind of texture regardless of differences in the point of view or suffered perspective distortion. The patches are computed using an affine covariant structure tensor defined at every pixel of the image domain, so that they can automatically adapt its shape and size. They are used in a segmentation model that uses an L1-norm fidelity term and fuzzy membership functions, which is solved by an alternating scheme. The output of the method is a partition of the image in regions with homogeneous texture together with a patch representative of the texture of each region.

This work presents an automatic method for optical flow inpainting. Given a video, each frame domain is endowed with a Riemannian metric based on the video pixel values. The missing optical flow is recovered by solving the Absolutely Minimizing Lipschitz Extension (AMLE) partial differential equation on the Riemannian manifold. An efficient numerical algorithm is proposed using eikonal operators for nonlinear elliptic partial differential equations on a finite graph. The choice of the metric is discussed and the method is applied to optical flow inpainting and sparse-to-dense optical flow estimation, achieving top-tier performance in terms of End-Point-Error (EPE).

We propose a new variational method for the completion of moving shapes through binary video inpainting that works by smoothly recovering the objects into an inpainting hole. We solve it by a simple dynamic shape analysis algorithm based on threshold dynamics. The model takes into account the optical flow and motion occlusions. The resulting inpainting algorithm diffuses the available information along the space and the visible trajectories of the pixels in time. We show its performance with examples from the Sintel dataset, which contains complex object motion and occlusions.

This paper presents an extension of the Non-Local Means denoising method, that effectively exploits the affine invariant self-similarities present in the images of real scenes. Our method provides a better image denoising result by grounding on the fact that in many occasions similar patches exist in the image but have undergone a transformation. The proposal uses an affine invariant patch similarity measure that performs an appropriate patch comparison by automatically and intrinsically adapting the size and shape of the patches. As a result, more similar patches are found and appropriately used. We show that this image denoising method achieves top-tier performance in terms of PSNR, outperforming consistently the results of the regular Non-Local Means, and that it provides state-of-the-art qualitative results.

In this paper we study the problem of comparing two patches of images defined on Riemannian manifolds which in turn can be defined by each image domain with a suitable metric depending on the image. For that we single out one particular instance of a set of models defining image similarities that was earlier studied in [C. Ballester et al., Multiscale Model. Simul., 12 (2014), pp. 616--649], using an axiomatic approach that extended the classical Álvarez--Guichard--Lions--Morel work to the nonlocal case. Namely, we study a linear model to compare patches defined on two images in $\mathbb{R}^N$ endowed with some metric. Besides its genericity, this linear model is selected by its computational feasibility since it can be approximated leading to an algorithm that has the complexity of the usual patch comparison using a weighted Euclidean distance. Moreover, we propose and study some intrinsic metrics which we define in terms of affine covariant structure tensors and we discuss their properties. These tensors are defined for any point in the image and are intrinsically endowed with affine covariant neighborhoods. We also discuss the effect of discretization over the affine covariance properties of the tensors. We illustrate our theoretical results with numerical experiments.

Using a classical result on algebraic invariants of the unimodular group, we present in this paper some basic geometric affine invariant quantities, and we use them to construct some distinctive descriptors for object detection. Although full affine invariance cannot be guaranteed due to noncommutativity of camera blur with affine maps and the domain problem (that is, the difficulty of finding an affine covariant domain), the proposed descriptors behave more robustly than SIFT with respect to affine deformations. This is supported by our comparisons both with the version of SIFT computed on an affine normalized neighborhood, and with ASIFT, which solves both the previously mentioned camera blur and domain problems by cleverly sampling the orbit of affine transformations of the images.

Video is usually acquired in interlaced format, where each image frame is composed of two image fields, each field holding same parity lines. However, many display devices require progressive video as input; also, many video processing tasks perform better on progressive material than on interlaced video. In the literature, there exist a great number of algorithms for interlaced to progressive video conversion, with a great tradeoff between the speed and quality of the results. The best algorithms in terms of image quality require motion compensation; hence, they are computationally very intensive. In this paper, we propose a novel deinterlacing algorithm based on ideas from the image inpainting arena. We view the lines to interpolate as gaps that we need to inpaint. Numerically, this is implemented using a dynamic programming procedure, which ensures a complexity of O(S), where S is the number of pixels in the image. The results obtained with our algorithm compare favorably, in terms of image quality, with state-of-the-art methods, but at a lower computational cost, since we do not need to perform motion field estimation.

In this paper we study a variational approach for filling-in regions of missing data in 2D and 3D digital images. Applications of this technique include the restoration of old photographs and removal of superimposed text like dates, subtitles, or publicity, or the zooming of images. The approach presented here, initially introduced in [C. Ballester et al., 2001], is based on a joint interpolation of the image gray-levels and gradient/isophotes directions, smoothly extending the isophote lines into the holes of missing data. The process underlying this approach can be considered as an interpretation of the Gestaltist's principle of good continuation. After discussing the model, we present the numerical algorithm used to minimize it and display some numerical experiments.

A variational approach for filling-in regions of missing data in digital images is introduced. The approach is based on joint interpolation of the image gray levels and gradient/isophotes directions, smoothly extending in an automatic fashion the isophote lines into the holes of missing data. This interpolation is computed by solving the variational problem via its gradient descent flow, which leads to a set of coupled second order partial differential equations, one for the gray-levels and one for the gradient orientations. The process underlying this approach can be considered as an interpretation of the Gestaltist's principle of good continuation. No limitations are imposed on the topology of the holes, and all regions of missing data can be simultaneously processed, even if they are surrounded by completely different structures. Applications of this technique include the restoration of old photographs and removal of superimposed text like dates, subtitles, or publicity. Examples of these applications are given. We conclude the paper with a number of theoretical results on the proposed variational approach and its corresponding gradient descent flow.

Inpainting, the technique of modifying an image in an undetectable form, is as ancient as art itself. The goals and applications of inpainting are numerous, from the restoration of damaged paintings and photographs to the removal/replacement of selected objects. In this paper, we introduce a novel algorithm for digital inpainting of still images that attempts to replicate the basic techniques used by professional restorators. After the user selects the regions to be restored, the algorithm automatically fills-in these regions with information surrounding them. The fill-in is done in such a way that isophote lines arriving at the regions' boundaries are completed inside. In contrast with previous approaches, the technique here introduced does not require the user to specify where the novel information comes from. This is automatically done (and in a fast way), thereby allowing to simultaneously fill-in numerous regions containing completely different structures and surrounding backgrounds. In addition, no limitations are imposed on the topology of the region to be inpainted. Applications of this technique include the restoration of old photographs and damaged film; removal of superimposed text like dates, subtitles, or publicity; and the removal of entire objects from the image like microphones or wires in special effects.