Deep learning has revolutionized the field of computer vision by introducing large scale neural networks with millions of parameters. Training these networks requires massive datasets and leads to intransparent models that can fail to generalize. At the other extreme, models designed from partial differential equations (PDEs) embed specialized domain knowledge into mathematical equations and usually rely on few manually chosen hyperparameters. This makes them transparent by construction and if designed and calibrated carefully, they can generalize well to unseen scenarios. In this paper, we show how to bring model- and data-driven approaches together by combining the explicit PDE-based approaches with convolutional neural networks to obtain the best of both worlds. We illustrate a joint architecture for the task of inpainting optical flow fields and show that the combination of model- and data-driven modeling leads to an effective architecture. Our model outperforms both fully explicit and fully data-driven baselines in terms of reconstruction quality, robustness and amount of required training data. Averaging the endpoint error across different mask densities, our method outperforms the explicit baselines by 11-27%, the GAN baseline by 47% and the Probabilisitic Diffusion baseline by 42%. With that, our method sets a new state of the art for inpainting of optical flow fields from random masks.

Abstract Diffusion-based inpainting can reconstruct missing image areas with high quality from sparse data, provided that their location and their values are well optimised. This is particularly useful for applications such as image compression, where the original image is known. Selecting the known data constitutes a challenging optimisation problem, that has so far been only investigated with model-based approaches. So far, these methods require a choice between either high quality or high speed since qualitatively convincing algorithms rely on many time-consuming inpaintings. We propose the first neural network architecture that allows fast optimisation of pixel positions and pixel values for homogeneous diffusion inpainting. During training, we combine two optimisation networks with a neural network-based surrogate solver for diffusion inpainting. This novel concept allows us to perform backpropagation based on inpainting results that approximate the solution of the inpainting equation. Without the need for a single inpainting during test time, our deep optimisation accelerates data selection by more than four orders of magnitude compared to common model-based approaches. This provides real-time performance with high quality results.

Inpainting-based image compression is emerging as a promising competitor to transform-based compression techniques. Its key idea is to reconstruct image information from only few known regions through inpainting. Specific partial differential equations (PDEs) such as edge-enhancing diffusion (EED) give high quality reconstructions of image structures with low or medium texture. Even though the strengths of PDE-and transform-based compression are complementary, they have rarely been combined within a hybrid codec. We propose to sparsify blocks of a JPEG compressed image and reconstruct them with EED inpainting. Our codec consistently outperforms JPEG and gives useful indications for successfully developing hybrid codecs further. Furthermore, our method is the first to choose regions rather than pixels as known data for PDE-based compression. It also gives novel insights into the importance of corner regions for EED-based codecs.

In spite of the fact that efficient compression methods for dense two-dimensional flow fields would be very useful for modern video codecs, hardly any research has been performed in this area so far. Our paper addresses this problem by proposing the first lossy diffusion-based codec for this purpose. It keeps only a few flow vectors on a coarse grid. Additionally stored edge locations ensure the accurate representation of discontinuities. In the decoding step, the missing information is recovered by homogeneous diffusion inpainting that incorporates the stored edges as reflecting boundary conditions. In spite of the simple nature of this codec, our experiments show that it achieves remarkable quality for compression ratios up to 800:1.

Inpainting-based compression has been suggested as a qualitative competitor to the JPEG family of transform-based codecs, specifically for high compression ratios. However, it also requires sophisticated interpolation, data optimisation and encoding tasks that are both slow and hard to implement. We propose a fast and simple alternative that combines Shepard interpolation with a novel joint inpainting and prediction approach. It represents the image by a fraction of its pixel values on a sparse regular subgrid that are selected by an efficient optimisation strategy. Experiments show that our codec is up to five orders of magnitude faster than traditional inpainting-based approaches. Qualitatively, it can surpass transform-based codecs, in particular for high compression ratios and cartoon-like images.

Inpainting techniques based on partial differential equations (PDEs), such as diffusion processes, are gaining growing importance as a novel family of image compression methods. Nevertheless, the application of inpainting in the field of hyperspectral imagery has been mainly focused on filling in missing information or dead pixels due to sensor failures. In this letter, we propose a novel PDE-based inpainting algorithm to compress hyperspectral images. The method inpaints separately the known data in the spatial and spectral dimensions. Then, it applies a prediction model to the final inpainting solution to obtain a representation much closer to the original image. Experimental results over a set of hyperspectral images indicate that the proposed algorithm can perform better than a recent proposed extension to prediction-based standard CCSDS-123.0 at low bit rate, better than JPEG 2000 Part 2 with the DWT 9/7 as a spectral transform at all bit rates, and competitive to JPEG 2000 with principal component analysis, the optimal spectral decorrelation transform for Gaussian sources.

The work of Levin et al. (2004) popularized stroke-based methods that add color to gray value images according to a small amount of user-specified color samples. Even though such reconstructions from sparse data suggest a possible use in compression, only few attempts were made so far in this direction. Diffusion-based compression methods pursue a similar idea: they store only few image pixels and inpaint the missing regions. Despite this close relation and a lack of diffusion-based color codecs, colorization ideas were so far only integrated into transform-based approaches such as JPEG. We address this missing link with two contributions. First, we show the relation between the discrete colorization of Levin et al. and continuous diffusion-based inpainting in the YCbCr color space. It decomposes the image into a luma (brightness) channel and two chroma (color) channels. Our luma-guided diffusion framework steers the diffusion inpainting in the chroma channels according to the structure in the luma channel. We show that making the luma-guided colorization anisotropic outperforms the method of Levin et al. significantly. Second, we propose a new luma preference codec that invests a large fraction of the bit budget into an accurate representation of the luma channel. This allows a high-quality reconstruction of color data with our colorization technique. Simultaneously, we exploit the fact that the human visual system is more sensitive to structural than to color information. Our experiments demonstrate that our new codec outperforms the state of the art in diffusion-based image compression and is competitive to transform-based codecs.

In image compression, codecs that rely on interpolation with partial differential equations (PDEs) are becoming increasingly popular. However, there have not been many attempts to transfer this concept to video compression. Since real-time performance is challenging for PDE-based reconstruction, first efficient approaches work on a frame-by-frame basis and focus on parallel implementations without considering coding quality. So far, there is no fully PDE-based video codec that exploits temporal redundancies. As a remedy, we propose a modular framework that combines PDE-based compression with motion compensation: Intra frames are predicted with PDE-based inpainting and inter frames with dense optic flow fields. We use this framework to develop a proof-of-concept codec that combines homogeneous diffusion inpainting with the variational optic flow model of Brox et al. (2004). Even without sophisticated parallelisation, we are able to perform real-time decompression of colour videos for the first time in PDE-based video compression.

Schmaltz et al. (2009) have shown that for reasonably high compression rates, diffusion-based codecs can exceed the quality of transformation-based methods such as JPEG 2000. They store only data at a few optimised pixel locations and in-paint missing data with edge-enhancing anisotropic diffusion (EED). However, research on compression with diffusion methods has mainly focussed on grey-value images, and colour images have been compressed in a straightforward way using anisotropic diffusion in RGB space. So far, there is no sophisticated diffusion-based counterpart to the colour mode of JPEG 2000. To address this shortcoming we introduce an advanced colour compression codec that exploits properties of the human visual system in YCbCr space. Since details in the luma channel Y are perceptually relevant, we invest a large fraction of our bit budget in its encoding with high fidelity. For the chroma channels Cb and Cr, the stored information can be very sparse, if we guide the EED-based inpainting with the high quality diffusion tensor from the luma reconstruction. Experiments demonstrate that our novel codec outperforms JPEG 2000 and compression with RGB-diffusion, both visually and quantitatively.