We introduce NeRD, a new demosaicking method for generating full-color images from Bayer patterns. Our approach leverages advancements in neural fields to perform demosaicking by representing an image as a coordinate-based neural network with sine activation functions. The inputs to the network are spatial coordinates and a low-resolution Bayer pattern, while the outputs are the corresponding RGB values. An encoder network, which is a blend of ResNet and U-net, enhances the implicit neural representation of the image to improve its quality and ensure spatial consistency through prior learning. Our experimental results demonstrate that NeRD outperforms traditional and state-of-the-art CNN-based methods and significantly closes the gap to transformer-based methods.

This paper proposes a novel approach to real-time automatic rim detection, classification, and inspection by combining traditional computer vision and deep learning techniques. At the end of every automotive assembly line, a quality control process is carried out to identify any potential defects in the produced cars. Common yet hazardous defects are related, for example, to incorrectly mounted rims. Routine inspections are mostly conducted by human workers that are negatively affected by factors such as fatigue or distraction. We have designed a new prototype to validate whether all four wheels on a single car match in size and type. Additionally, we present three comprehensive open-source databases, CWD1500, WHEEL22, and RB600, for wheel, rim, and bolt detection, as well as rim classification, which are free-to-use for scientific purposes.

Abstract Blur is an image degradation that makes object recognition challenging. Restoration approaches solve this problem via image deblurring, deep learning methods rely on the augmentation of training sets. Invariants with respect to blur offer an alternative way of describing and recognising blurred images without any deblurring and data augmentation. In this paper, we present an original theory of blur invariants. Unlike all previous attempts, the new theory requires no prior knowledge of the blur type. The invariants are constructed in the Fourier domain by means of orthogonal projection operators and moment expansion is used for efficient and stable computation. Applying a general substitution rule, combined invariants to blur and spatial transformations are easy to construct and use. Experimental comparison to Convolutional Neural Networks shows the advantages of the proposed theory.

Structured illumination microscopy is a widely popular super-resolution technique for live cell imaging capable of surpassing the diffraction limit. Its temporal resolution is limited by the need to capture multiple low-resolution images to reconstruct a single high-resolution image. When observing rapid biological processes, the local movement between frames leads to the formation of reconstruction artifacts, which subsequently impair the data interpretation. We propose to include this type of movement in the definition of the image formation forward problem. The motion can then be estimated from the original data using optical flow, and the optimization problem is solved using the alternating direction method of multipliers. Our approach is tested against other reconstruction techniques on both synthetic and real biological data.

We propose the first learning-based approach for fast moving objects detection. Such objects are highly blurred and move over large distances within one video frame. Fast moving objects are associated with a deblurring and matting problem, also called deblatting. We show that the separation of deblatting into consecutive matting and deblurring allows achieving real-time performance, i.e. an order of magnitude speed-up, and thus enabling new classes of application. The proposed method detects fast moving objects as a truncated distance function to the trajectory by learning from synthetic data. For the sharp appearance estimation and accurate trajectory estimation, we propose a matting and fitting network that estimates the blurred appearance without background, followed by an energy minimization based deblurring. The state-of-the-art methods are outperformed in terms of recall, precision, trajectory estimation, and sharp appearance reconstruction. Compared to other methods, such as deblatting, the inference is of several orders of magnitude faster and allows applications such as real-time fast moving object detection and retrieval in large video collections.

The field of blind image deblurring was for a long time dominated by Maximum-A-Posteriori methods seeking the optimal pair of sharp image-blur of a suitable functional. Recently, learning-based methods, especially those based on deep convolutional neural networks, are proving effective and are receiving increasing attention by the research community. In 2020, Ren et al. proposed a deblurring method called SelfDeblur which combines the model-driven approach of traditional MAP methods and the generative power of neural nets. The method is capable of producing very high-quality results, yet it inherits some problems of MAP methods, especially possible convergence to a wrong local optimum. In this paper we propose several easy-to-implement modifications of SelfDeblur, namely suitable initialization, multiscale processing, and regularization, that improve the average performance of the original method and decrease the probability of failure.

Abstract Objects moving at high speed along complex trajectories often appear in videos, especially videos of sports. Such objects travel a considerable distance during exposure time of a single frame, and therefore, their position in the frame is not well defined. They appear as semi-transparent streaks due to the motion blur and cannot be reliably tracked by general trackers. We propose a novel approach called Tracking by Deblatting based on the observation that motion blur is directly related to the intra-frame trajectory of an object. Blur is estimated by solving two intertwined inverse problems, blind deblurring and image matting, which we call deblatting. By postprocessing, non-causal Tracking by Deblatting estimates continuous, complete, and accurate object trajectories for the whole sequence. Tracked objects are precisely localized with higher temporal resolution than by conventional trackers. Energy minimization by dynamic programming is used to detect abrupt changes of motion, called bounces. High-order polynomials are then fitted to smooth trajectory segments between bounces. The output is a continuous trajectory function that assigns location for every real-valued time stamp from zero to the number of frames. The proposed algorithm was evaluated on a newly created dataset of videos from a high-speed camera using a novel Trajectory-IoU metric that generalizes the traditional Intersection over Union and measures the accuracy of the intra-frame trajectory. The proposed method outperforms the baselines both in recall and trajectory accuracy. Additionally, we show that from the trajectory function precise physical calculations are possible, such as radius, gravity, and sub-frame object velocity. Velocity estimation is compared to the high-speed camera measurements and radars. Results show high performance of the proposed method in terms of Trajectory-IoU, recall, and velocity estimation.

This article is a comment on the recent TPAMI paper (Gopalan et al., 2012) that introduced a blur-invariant distance measure between two images. We point out two mistakes of the theory presented in (Gopalan et al., 2012) and propose a correction. We also compare the original and corrected methods experimentally.

We propose a novel method that tracks fast moving objects, mainly non-uniform spherical, in full 6 degrees of freedom, estimating simultaneously their 3D motion trajectory, 3D pose and object appearance changes with a time step that is a fraction of the video frame exposure time. The sub-frame object localization and appearance estimation allows realistic temporal super-resolution and precise shape estimation. The method, called TbD-3D (Tracking by Deblatting in 3D) relies on a novel reconstruction algorithm which solves a piece-wise deblurring and matting problem. The 3D rotation is estimated by minimizing the reprojection error. As a second contribution, we present a new challenging dataset with fast moving objects that change their appearance and distance to the camera. High-speed camera recordings with zero lag between frame exposures were used to generate videos with different frame rates annotated with ground-truth trajectory and pose.

Deep learning-based methods for classification and segmentation require large training sets. Generating training data is often a tedious and expensive task. In industrial applications, such as automated visual inspection of products in an assemble line, objects for classification are well defined yet labeled data are difficult to obtain. To alleviate the problem of manual labeling, we propose to train a convolutional neural network with an automatically generated training set using a naive classifier with handcrafted features. We show that when the naive classifier has high precision then the trained network has both high precision and recall despite the low recall of the naive classifier. We demonstrate the proposed methodology on real scenario of detecting a car coolant tank. However, the proposed methodology facilitates collection of train data for a wider type of CNN based methods such as near-duplicate image detection or segmenting tampered areas of images.

Priors play an important role of regularizers in image deblurring algorithms. Image priors are frequently studied and many forms were proposed in the literature. Blur priors are considered less important and the most common forms are simple uniform distributions with domain constraints. We propose a more informative blur prior based on the notion of atomic norm which favors blurs composed of line segments and is suitable for motion blur. The prior is formulated as a linear program that can be inserted into any optimization task. Evaluation is conducted on blind deblurring of moving objects.

Images acquired with standard digital cameras have Bayer patterns and suffer from lens blur. A demosaicking step is implemented in every digital camera, yet blur often remains unattended due to computational cost and instability of deblur-ring algorithms. Linear methods, which are computationally less demanding, produce ringing artifacts in deblurred images. Complex non-linear deblurring methods avoid artifacts, however their complexity imply offline application after camera demosaicking, which leads to sub-optimal performance. In this work, we propose a joint demosaicking deblurring and deringing network with a light-weight architecture inspired by the alternating direction method of multipliers. The proposed network has a transparent and clear interpretation compared to other black-box data driven approaches. We experimentally validate its superiority over state-of-the-art demosaicking methods with offline deblurring.

Tracking Fast Moving Objects (FMO), which appear as blurred streaks in video sequences, is a difficult task for standard trackers, as the object position does not overlap in consecutive video frames and texture information of the objects is blurred. Up-to-date approaches tuned for this task are based on background subtraction with a static background and slow deblurring algorithms. In this article, we present a tracking-by-segmentation approach implemented using modern deep learning methods that perform near real-time tracking on real-world video sequences. We have developed a physically plausible FMO sequence generator to be a robust foundation for our training pipeline and demonstrate straightforward network adaptation for different FMO scenarios with varying foreground.

If an object is photographed at motion in front of a static background, the object will be blurred while the background sharp and partially occluded by the object. The goal is to recover the object appearance from such blurred image. We adopt the image formation model for fast moving objects and consider objects undergoing 2D translation and rotation. For this scenario we formulate the estimation of the object shape, appearance, and motion from a single image and known background as a constrained optimization problem with appropriate regularization terms. Both similarities and differences with blind deconvolution are discussed with the latter caused mainly by the coupling of the object appearance and shape in the acquisition model. Necessary conditions for solution uniqueness are derived and a numerical solution based on the alternating direction method of multipliers is presented. The proposed method is evaluated on a new dataset.

Image deblurring is one of the standard problems in image processing. Recently, this area of research is dominated by blind deblurring, where neither the sharp image nor the blur are known. The majority of works, however, target scenarios where the captured scene is static and the blur is caused by camera motion, i.e. the whole image is blurred. In this work we address a similar yet different scenario: an object moves in front of a static background. Such object is blurred due to motion while the background is sharp and partially occluded by the object. The problem of blind deblurring in such setting has not been properly addressed in literature. We formally define the problem, discuss its solvability, and explain why it cannot be viewed as a special case of classical blind de-blurring. We propose a solution to the presented problem for a particular class of motions and demonstrate results on real data.

Image restoration is typically defined as an ill-posed problem which has to be regularized to obtain an acceptable solution. In Bayesian interpretation, regularization is equivalent to prior model of the image. An added value of Bayesian point of view is the ability to form a hierarchical model and estimate the hyperparameters of the prior from the data. Many prior models are available, usually based on automatic relevance determination principle applied to the transformed image. However, the transformation (the most common is a differential operator) is assumed to be known. In this letter, we propose to relax this assumption and estimate the image transformation from the data. The resulting algorithm is analytically tractable using the variational Bayes method. Properties of the new prior are demonstrated on the problem of image superresolution.

The notion of a Fast Moving Object (FMO), i.e. an object that moves over a distance exceeding its size within the exposure time, is introduced. FMOs may, and typically do, rotate with high angular speed. FMOs are very common in sports videos, but are not rare elsewhere. In a single frame, such objects are often barely visible and appear as semitransparent streaks. A method for the detection and tracking of FMOs is proposed. The method consists of three distinct algorithms, which form an efficient localization pipeline that operates successfully in a broad range of conditions. We show that it is possible to recover the appearance of the object and its axis of rotation, despite its blurred appearance. The proposed method is evaluated on a new annotated dataset. The results show that existing trackers are inadequate for the problem of FMO localization and a new approach is required. Two applications of localization, temporal superresolution and highlighting, are presented.

Blind deconvolution is a strongly ill-posed problem comprising of simultaneous blur and image estimation. Recent advances in prior modeling and/or inference methodology led to methods that started to perform reasonably well in real cases. However, as we show here, they tend to fail if the convolution model is violated even in a small part of the image. Methods based on variational Bayesian inference play a prominent role. In this paper, we use this inference in combination with the same prior for noise, image, and blur that belongs to the family of independent non-identical Gaussian distributions, known as the automatic relevance determination prior. We identify several important properties of this prior useful in blind deconvolution, namely, enforcing non-negativity of the blur kernel, favoring sharp images over blurred ones, and most importantly, handling non-Gaussian noise, which, as we demonstrate, is common in real scenarios. The presented method handles discrepancies in the convolution model, and thus extends applicability of blind deconvolution to real scenarios, such as photos blurred by camera motion and incorrect focus.

Standard convolution as a model of radiometric degradation is in majority of cases inaccurate as the blur varies in space and we are thus required to work with a computationally demanding space-variant model. Space-variant degradation can be approximately decomposed to a set of standard convolutions. We explain in detail the properties of the space-variant degradation operator and show two possible decomposition models and two approximation approaches. Our target application is space-variant image deconvolution, on which we illustrate theoretical differences between these models. We propose a computationally efficient restoration algorithm that belongs to a category of alternating direction methods of multipliers, which consists of four update steps with closed-form solutions. Depending on the used decomposition, two variations of the algorithm exist with distinct properties. We test the effectiveness of the decomposition models under different levels of approximation on synthetic and real examples, and conclude the letter by drawing several practical observations.

In this work we address the problem of blind deconvolution and denoising. We focus on restoration of text documents and we show that this type of highly structured data can be successfully restored by a convolutional neural network. The networks are trained to reconstruct high-quality images directly from blurry inputs without assuming any specific blur and noise models. We demonstrate the performance of the convolutional networks on a large set of text documents and on a combination of realistic de-focus and camera shake blur kernels. On this artificial data, the convolutional networks significantly outperform existing blind deconvolution methods, including those optimized for text, in terms of image quality and OCR accuracy. In fact, the networks outperform even state-of-the-art non-blind methods for anything but the lowest noise levels. The approach is validated on real photos taken by various devices.

Given the large amount of blur estimation and blind deconvolution methods just in the last decade, there is an increasing need to compare the performance of a particular method with others. Unlike in other fields in image processing, there are very few well-established benchmark databases of test data and, more importantly, no standard way of performance evaluation. In this paper, we focus on the latter. We propose a new error measure for the blur kernel - a method for comparison of the blur estimate with the ground truth - which correctly reflects how inaccuracies in the blur estimation affect the subsequent image restoration, without the necessity to perform the actual deconvolution.

Videokymography (VKG) is a high-speed medical imaging technique used in laryngology and phoniatrics for examination of vocal fold vibrations, it offers important characteristics for diagnosis and treatment of voice disorders. VKG repeatedly scans only a single line from the scene and captures movements of vocal folds in this region of interest. This paper proposes methods for computer assisted evaluation of diagnostically important vibration features, related to movements of vocal folds and their surroundings. They are derived from existing as well as newly developed methods of digital image processing, mainly based on data segmentation and morphological operations. Performance of the developed methods is compared to expert manual assessments and it proves to be comparable with clinicians conclusions.

Smartphones are now widely used as photographic devices. Equipped with cheap cameras they are prone to many degradations, most notably handshake in combination with rolling shutter causes severe space-variant blur. Removing blur without any information about the camera motion is a computationally demanding and unstable process. We use built-in gyroscopes to record the motion trajectory of the camera during exposure and then remove blur from the acquired photograph based on the reconstructed trajectory. The proposed deblurring application is implemented on Android smartphones with close-to-real-time performance.

Removing blurs from a single degraded image without any knowledge of the blur kernel is an ill-posed blind deconvolution problem. Proper estimators together with correct image priors play a fundamental role in accurate blind de-convolution. We demonstrate a superior performance of the variational Bayesian estimator and discuss suitability of automatic relevance determination distributions as image priors. Restoration of real photos blurred by out-of-focus and motion blur, and comparison with a state-of-the-art method is provided.

We propose a method for removal of space-variant blur from images predominantly degraded by camera shake without any knowledge of camera trajectory. Blurs are first estimated in a small number of image patches. We derive a novel parametric blur interpolation method and discuss conditions under which it can be used to exactly calculate blurs for every pixel position. Having this information, we restore the sharp image by a standard regularization technique. Performance of the proposed method is experimentally validated.

Blind deconvolution, which comprises simultaneous blur and image estimations, is a strongly ill-posed problem. It is by now well known that if multiple images of the same scene are acquired, this multichannel (MC) blind deconvolution problem is better posed and allows blur estimation directly from the degraded images. We improve the MC idea by adding robustness to noise and stability in the case of large blurs or if the blur size is vastly overestimated. We formulate blind deconvolution as an l(1) -regularized optimization problem and seek a solution by alternately optimizing with respect to the image and with respect to blurs. Each optimization step is converted to a constrained problem by variable splitting and then is addressed with an augmented Lagrangian method, which permits simple and fast implementation in the Fourier domain. The rapid convergence of the proposed method is illustrated on synthetically blurred data. Applicability is also demonstrated on the deconvolution of real photos taken by a digital camera.

We propose a fast algorithm for solving the inverse problem of resolution enhancement (superresolution). Robustness is achieved by a non-linear regularizer and a method based on variable splitting is used to obtain an equivalent linear formulation. Special attention is paid to fast implementation using the Fourier transform. In particular, we show that a degradation operator (downsampling) can be implemented in the frequency domain and that all computations can be performed very efficiently without losing robustness. To our knowledge, this is the first attempt towards a very fast SR algorithm, which retains favorable edge-preserving properties of non-linear regularizers.

We propose a practical method to remove photo blur due to camera shake, which is a typical problem when taking photos in dim lighting conditions such as indoor or night scenes. We use a pair of images, one of them blurred and the other one underexposed or noisy because of high ISO setting. Existing methods assume convolution model, that is the same blur in the whole image. It is seldom true in practice, especially for wide angle lens photos. We apply a space-variant model of blurring valid in many real situations. Results are documented by a photograph of a night scene.

Functional properties of living tissues appear in PET, whereas structural information at significantly higher resolution and better image quality is provided by other modalities, such as CT or MRI. We illustrate how structural information of matched anatomic images can be used as priors in the total variation denoising and blind deconvolution of functional PET images. Experiments on phantom images and clinical data validate the proposed method.

In many real applications, traditional super-resolution (SR) methods fail to provide high-resolution images due to objectionable blur and inaccurate registration of input low-resolution images. Only integer resolution enhancement factors, such as 2 or 3, are often considered, but non-integer factors between 1 and 2 are also important in real cases. We introduce a method to SR and deconvolution, which assumes no prior information about the shape of degradation blurs, incorporates registration parameters, and is properly defined for any rational (fractional) resolution factor. The method minimizes a regularized energy function with respect to the high-resolution image and blurs, where regularization is carried out in both the image and blur domains. The blur regularization is based on a generalized multi-channel blind deconvolution constraint derived in the paper. An extension to color images is briefly discussed. Experiments on real data illustrate robustness to noise and other advantages of the method.

In many real applications traditional superresolution methods fail to provide high-resolution images due to objectionable blur and inaccurate registration of input low-resolution images. In this paper, we present a method of superresolution and blind deconvolution of video sequences and address problems of misregistration, local motion and change of illumination. The method processes the video by applying temporal windows, masking out regions of misregistration, and minimizing a regularized energy function with respect to the high-resolution frame and blurs, where regularization is carried out in both the image and blur domains. Experiments on real video sequences illustrate robustness of the method.

This paper presents a new approach to the blind deconvolution and superresolution problem of multiple degraded low-resolution frames of the original scene. We do not assume any prior information about the shape of degradation blurs. The proposed approach consists of building a regularized energy function and minimizing it with respect to the original image and blurs, where regularization is carried out in both the image and blur domains. The image regularization based on variational principles maintains stable performance under severe noise corruption. The blur regularization guarantees consistency of the solution by exploiting differences among the acquired low-resolution images. Several experiments on synthetic and real data illustrate the robustness and utilization of the proposed technique in real applications.

Existing multichannel blind restoration techniques assume perfect spatial alignment of channels, correct estimation of blur size, and are prone to noise. We developed an alternating minimization scheme based on a maximum a posteriori estimation with a priori distribution of blurs derived from the multichannel framework and a priori distribution of original images defined by the variational integral. This stochastic approach enables us to recover the blurs and the original image from channels severely corrupted by noise. We observe that the exact knowledge of the blur size is not necessary, and we prove that translation misregistration up to a certain extent can be automatically removed in the restoration process.

Blind image deconvolution is required in many applications of microscopy imaging, remote sensing, and astronomical imaging. Unfortunately in a single-channel framework, serious conceptual and numerical problems are often encountered. Very recently, an eigenvector-based method (EVAM) was proposed for a multichannel framework which determines perfectly convolution masks in a noise-free environment if channel disparity, called co-primeness, is satisfied. We propose a novel iterative algorithm based on recent anisotropic denoising techniques of total variation and a Mumford-Shah functional with the EVAM restoration condition included. A linearization scheme of half-quadratic regularization together with a cell-centered finite difference discretization scheme is used in the algorithm and provides a unified approach to the solution of total variation or Mumford-Shah. The algorithm performs well even on very noisy images and does not require an exact estimation of mask orders. We demonstrate capabilities of the algorithm on synthetic data. Finally, the algorithm is applied to defocused images taken with a digital camera and to data from astronomical ground-based observations of the Sun.

We present an application of digital image processing to the analysis of the conservation of a medieval mosaic. The art piece is the "The Last Judgement" mosaic, situated on the wall of St Vitus Cathedral in Prague, Czech Republic. A 19/sup th/ century historical photograph of the mosaic was compared with a photograph of its current reconstructed state. The images were first preprocessed to increase their quality (noise reduction, deblurring). In the second stage, geometrical differences between images were removed by means of image registration techniques. Finally, differences between the current and historical photographs were identified.

We present a multichannel blind deconvolution method based on so-called subspace technique that was originally proposed by Harikumar and Bresler (1996, 1999). When at least two differently degraded images (channels) of the original scene are provided, the method is better conditioned than classical single channel ones. In comparison with earlier multichannel blind deconvolution techniques the subspace method is not iterative and this possibly implies an implementation that can be computationally more efficient. An application of the proposed method to the restoration of the images of sunspots is presented.