In this paper, we present ECSIC, a novel learned method for stereo image compression. Our proposed method compresses the left and right images in a joint manner by exploiting the mutual information between the images of the stereo image pair using a novel stereo cross attention (SCA) module and two stereo context modules. The SCA module performs cross-attention restricted to the corresponding epipolar lines of the two images and processes them in parallel. The stereo context modules improve the entropy estimation of the second encoded image by using the first image as a context. We conduct an extensive ablation study demonstrating the effectiveness of the proposed modules and a comprehensive quantitative and qualitative comparison with existing methods. ECSIC achieves state-of-the-art performance in stereo image compression on the two popular stereo image datasets Cityscapes and InStereo2k while allowing for fast encoding and decoding.

We propose a learned method for stereo image compression that leverages the similarity of the left and right images in a stereo pair due to overlapping fields of view. The left image is compressed by a learned compression method based on an autoencoder with a hyperprior entropy model. The right image uses this information from the previously encoded left image in both the encoding and decoding stages. In particular, for the right image, we encode only the residual of its latent representation to the optimally shifted latent of the left image. On top of that, we also employ a stereo attention module to connect left and right images during decoding. The performance of the proposed method is evaluated on two benchmark stereo image datasets (Cityscapes and InStereo2K) and outperforms previous stereo image compression methods while being significantly smaller in model size.

The field of blind image deblurring was for a long time dominated by Maximum-A-Posteriori methods seeking the optimal pair of sharp image-blur of a suitable functional. Recently, learning-based methods, especially those based on deep convolutional neural networks, are proving effective and are receiving increasing attention by the research community. In 2020, Ren et al. proposed a deblurring method called SelfDeblur which combines the model-driven approach of traditional MAP methods and the generative power of neural nets. The method is capable of producing very high-quality results, yet it inherits some problems of MAP methods, especially possible convergence to a wrong local optimum. In this paper we propose several easy-to-implement modifications of SelfDeblur, namely suitable initialization, multiscale processing, and regularization, that improve the average performance of the original method and decrease the probability of failure.

Abstract Objects moving at high speed along complex trajectories often appear in videos, especially videos of sports. Such objects travel a considerable distance during exposure time of a single frame, and therefore, their position in the frame is not well defined. They appear as semi-transparent streaks due to the motion blur and cannot be reliably tracked by general trackers. We propose a novel approach called Tracking by Deblatting based on the observation that motion blur is directly related to the intra-frame trajectory of an object. Blur is estimated by solving two intertwined inverse problems, blind deblurring and image matting, which we call deblatting. By postprocessing, non-causal Tracking by Deblatting estimates continuous, complete, and accurate object trajectories for the whole sequence. Tracked objects are precisely localized with higher temporal resolution than by conventional trackers. Energy minimization by dynamic programming is used to detect abrupt changes of motion, called bounces. High-order polynomials are then fitted to smooth trajectory segments between bounces. The output is a continuous trajectory function that assigns location for every real-valued time stamp from zero to the number of frames. The proposed algorithm was evaluated on a newly created dataset of videos from a high-speed camera using a novel Trajectory-IoU metric that generalizes the traditional Intersection over Union and measures the accuracy of the intra-frame trajectory. The proposed method outperforms the baselines both in recall and trajectory accuracy. Additionally, we show that from the trajectory function precise physical calculations are possible, such as radius, gravity, and sub-frame object velocity. Velocity estimation is compared to the high-speed camera measurements and radars. Results show high performance of the proposed method in terms of Trajectory-IoU, recall, and velocity estimation.

We propose a novel method that tracks fast moving objects, mainly non-uniform spherical, in full 6 degrees of freedom, estimating simultaneously their 3D motion trajectory, 3D pose and object appearance changes with a time step that is a fraction of the video frame exposure time. The sub-frame object localization and appearance estimation allows realistic temporal super-resolution and precise shape estimation. The method, called TbD-3D (Tracking by Deblatting in 3D) relies on a novel reconstruction algorithm which solves a piece-wise deblurring and matting problem. The 3D rotation is estimated by minimizing the reprojection error. As a second contribution, we present a new challenging dataset with fast moving objects that change their appearance and distance to the camera. High-speed camera recordings with zero lag between frame exposures were used to generate videos with different frame rates annotated with ground-truth trajectory and pose.

Priors play an important role of regularizers in image deblurring algorithms. Image priors are frequently studied and many forms were proposed in the literature. Blur priors are considered less important and the most common forms are simple uniform distributions with domain constraints. We propose a more informative blur prior based on the notion of atomic norm which favors blurs composed of line segments and is suitable for motion blur. The prior is formulated as a linear program that can be inserted into any optimization task. Evaluation is conducted on blind deblurring of moving objects.

If an object is photographed at motion in front of a static background, the object will be blurred while the background sharp and partially occluded by the object. The goal is to recover the object appearance from such blurred image. We adopt the image formation model for fast moving objects and consider objects undergoing 2D translation and rotation. For this scenario we formulate the estimation of the object shape, appearance, and motion from a single image and known background as a constrained optimization problem with appropriate regularization terms. Both similarities and differences with blind deconvolution are discussed with the latter caused mainly by the coupling of the object appearance and shape in the acquisition model. Necessary conditions for solution uniqueness are derived and a numerical solution based on the alternating direction method of multipliers is presented. The proposed method is evaluated on a new dataset.

Image deblurring is one of the standard problems in image processing. Recently, this area of research is dominated by blind deblurring, where neither the sharp image nor the blur are known. The majority of works, however, target scenarios where the captured scene is static and the blur is caused by camera motion, i.e. the whole image is blurred. In this work we address a similar yet different scenario: an object moves in front of a static background. Such object is blurred due to motion while the background is sharp and partially occluded by the object. The problem of blind deblurring in such setting has not been properly addressed in literature. We formally define the problem, discuss its solvability, and explain why it cannot be viewed as a special case of classical blind de-blurring. We propose a solution to the presented problem for a particular class of motions and demonstrate results on real data.

The notion of a Fast Moving Object (FMO), i.e. an object that moves over a distance exceeding its size within the exposure time, is introduced. FMOs may, and typically do, rotate with high angular speed. FMOs are very common in sports videos, but are not rare elsewhere. In a single frame, such objects are often barely visible and appear as semitransparent streaks. A method for the detection and tracking of FMOs is proposed. The method consists of three distinct algorithms, which form an efficient localization pipeline that operates successfully in a broad range of conditions. We show that it is possible to recover the appearance of the object and its axis of rotation, despite its blurred appearance. The proposed method is evaluated on a new annotated dataset. The results show that existing trackers are inadequate for the problem of FMO localization and a new approach is required. Two applications of localization, temporal superresolution and highlighting, are presented.

Blind deconvolution is a strongly ill-posed problem comprising of simultaneous blur and image estimation. Recent advances in prior modeling and/or inference methodology led to methods that started to perform reasonably well in real cases. However, as we show here, they tend to fail if the convolution model is violated even in a small part of the image. Methods based on variational Bayesian inference play a prominent role. In this paper, we use this inference in combination with the same prior for noise, image, and blur that belongs to the family of independent non-identical Gaussian distributions, known as the automatic relevance determination prior. We identify several important properties of this prior useful in blind deconvolution, namely, enforcing non-negativity of the blur kernel, favoring sharp images over blurred ones, and most importantly, handling non-Gaussian noise, which, as we demonstrate, is common in real scenarios. The presented method handles discrepancies in the convolution model, and thus extends applicability of blind deconvolution to real scenarios, such as photos blurred by camera motion and incorrect focus.

In this work we address the problem of blind deconvolution and denoising. We focus on restoration of text documents and we show that this type of highly structured data can be successfully restored by a convolutional neural network. The networks are trained to reconstruct high-quality images directly from blurry inputs without assuming any specific blur and noise models. We demonstrate the performance of the convolutional networks on a large set of text documents and on a combination of realistic de-focus and camera shake blur kernels. On this artificial data, the convolutional networks significantly outperform existing blind deconvolution methods, including those optimized for text, in terms of image quality and OCR accuracy. In fact, the networks outperform even state-of-the-art non-blind methods for anything but the lowest noise levels. The approach is validated on real photos taken by various devices.

Given the large amount of blur estimation and blind deconvolution methods just in the last decade, there is an increasing need to compare the performance of a particular method with others. Unlike in other fields in image processing, there are very few well-established benchmark databases of test data and, more importantly, no standard way of performance evaluation. In this paper, we focus on the latter. We propose a new error measure for the blur kernel - a method for comparison of the blur estimate with the ground truth - which correctly reflects how inaccuracies in the blur estimation affect the subsequent image restoration, without the necessity to perform the actual deconvolution.

Removing blurs from a single degraded image without any knowledge of the blur kernel is an ill-posed blind deconvolution problem. Proper estimators together with correct image priors play a fundamental role in accurate blind de-convolution. We demonstrate a superior performance of the variational Bayesian estimator and discuss suitability of automatic relevance determination distributions as image priors. Restoration of real photos blurred by out-of-focus and motion blur, and comparison with a state-of-the-art method is provided.