Joint denoising and demosaicing (JDD) aims to reconstruct high-quality color images from noisy RAW measurements. Recent deep learning approaches often rely on monolithic architectures with high computational cost. We propose JDD-LI, a lightweight modular JDD pipeline designed to improve the accuracy–efficiency trade-off. The method decomposes the task into three stages aligned with a simplified image signal processing pipeline: RAW-domain denoising, pretrained demosaicing, and RGB-domain denoising. The model is trained using a two-stage strategy where the demosaicer is first optimized under noise-free conditions and then fixed while the denoising modules are trained end-to-end. This design stabilizes training and simplifies optimization by decoupling the demosaicing and denoising stages. Experiments on Synthetic RAW and Mosaicked RGB benchmarks show competitive reconstruction quality with reduced computational cost. The modular structure also supports process-level interpretability through physically meaningful intermediate representations and ablation analysis.

Denoising is one of the fundamental steps of the processing pipeline that converts data captured by a camera sensor into a display-ready image or video. It is generally performed early in the pipeline, usually before demosaicking, although studies swapping their order or even conducting them jointly have been proposed. With the advent of deep learning, the quality of denoising algorithms has steadily increased. Even so, modern neural networks still have a hard time adapting to new noise levels and scenes, which is indispensable for real-world applications. With those in mind, we propose a self-similarity-based denoising scheme that weights both a pre- and a post-demosaicking denoiser for Bayer-patterned CFA video data. We show that a balance between the two leads to better image quality, and we empirically find that higher noise levels benefit from a higher influence pre-demosaicking. We also integrate temporal trajectory prefiltering steps before each denoiser, which further improve texture reconstruction. The proposed method only requires an estimation of the noise model at the sensor, accurately adapts to any noise level, and is competitive with the state of the art, making it suitable for real-world videography.

High dynamic range (HDR) video synthesis is a very challenging task. Consecutive frames are acquired with alternate expositions, generally two or three different exposure times. Classical methods aim at registering neighbouring frames and fuse them using image HDR techniques. However, the registration often fails to obtain accurate results and the fusion produces ghosting artifacts. Deep learning techniques have recently appeared imitating the structure of existing classical methods. The neural network is intended to estimate the registration function and choose the fusion weights. In this paper, we propose a new method for HDR video synthesis using a variational model. The proposed model uses a nonlocal regularization term to combine pixel information from neighbouring frames. The obtained results are competitive with state-of-the-art. Moreover, the proposed method gives a more reliable and understandable solution than deep-learning based ones.

Local patch-based algorithms for image registration fail to accurately match points in areas not discriminative enough, mainly textureless regions. These methods normally involve a validation process and provide a non-completely dense solution. In this paper, we propose a novel refinement and completion approach for registration. The proposed model combines single image nonlocal densification with classical variational image registration. We associate a total variation regularization with a nonlocal term to provide a smooth solution leveraging the image geometry. We show experiments on public stereo and optical flow datasets to filter and densify incomplete depth maps and motion fields. Extensive comparisons against existing and state-of-the-art depth/motion fields densification approaches demonstrate the competitive performance of the introduced method. Additionally, we illustrate how our method can deal with other tasks, such as filtering and interpolation of depth maps from RGBD data and depth upsampling.

The authors present a method for the enhancement of backlit images, i.e. images in which the main source of light is behind the photography subject. These images contain, simultaneously, very dark and very bright regions. In this situation, a single tone mapping function is unable to enhance the whole image. They propose the use of several such tone mappings, some of them enhancing the dark regions while others enhancing the bright regions, and then the combination of all these results using an image fusion algorithm. Qualitative and quantitative results confirm the validity of the proposed method.

Demosaicking and denoising are key steps in the camera imaging chain for both images and videos. The reconstruction errors during these stages will have undesirable effects on the final result if not handled properly. Demosaicking provokes the spatial and color correlation of noise, which is afterwards enhanced by the processing pipeline. This structured noise generally degrades the image quality and, for dark scenes with low signal to noise ratio, prevents the correct interpretation of the image. When trying to mitigate such structured noise on already processed data, denoising methods attenuate details and texture. We present a video processing chain, consisting of a novel strategy for the removal of noise at the camera sensor and a novel video demosaicking algorithm. In both cases, a spatio-temporal patch-based filter with motion compensation is introduced. The experimental results, including real examples, illustrate the performance of the proposed chain, avoiding the creation of interpolation artefacts and colored spots.

A new method for denoising video sequences is presented. The method is able to deal with poor quality sequences affected by both noise and compression, as for example mobile phone videos recorded in low-light conditions. In this real scenario, the usual white Gaussian uniform noise assumption fails, and the state-of-the art denoising algorithms are ineffective. The proposed method first estimates a signal dependent noise model at each level of a multi-scale pyramid; then, a variance stabilization transform is applied at each scale; and, finally, a denoising algorithm is applied on the stabilized image sequence. This algorithm takes advantage of self similarity and redundancy of adjacent frames and uses motion compensation by regularized optical flow methods, which permits robust patch comparison in a spatio-temporal volume. The experiments illustrate that the proposed method is able to correctly remove highly correlated noise from real dark and compressed movie sequences. Finally, in order to assess numerically the performance of the proposed algorithm, we introduce a new database of noisy and compressed image sequences with known ground truth. The numerical comparisons with the introduced database corroborate the superior performance of the method.

Pansharpening techniques aim at fusing a low-spatial resolution multispectral (MS) image with a higher spatial resolution panchromatic (PAN) image to produce an MS image at high spatial resolution. Despite significant progress in the field, spectral and spatial distortions might still compromise the quality of the results. We introduce a restoration strategy to mitigate artifacts of fused products. After applying the principal component analysis transform to a pansharpened image, the chromatic components are filtered conditionally to the geometry of PAN. The structural component is then replaced by the locally histogram-matched PAN for spatial enhancement. Experimental results illustrate the efficiency of the proposed restoration chain.

The demosaicking provokes the spatial and color correlation of noise, which is afterwards enhanced by the imaging pipeline. The correct removal previous or simultaneously with the demosaicking process is not usually considered in the literature. We present a novel joint demosaicking and denoising algorithm for image sequences. The proposed algorithm uses a spatio-temporal patch method modifying all pixels, including those of the Bayer CFA. However, only original values are considered for averaging. The experimentation, including real examples, illustrates how a joint denoising and demosaicking algorithm avoids the creation of artifacts and colored spots in the final image.

Local algorithms for stereo fail to match accurately points in areas not discriminative enough, mainly texture-less regions. We propose a method for filtering incorrect depth estimates and filling in those areas which have not been matched. The proposed model combines total variation regularization with a non local term taking advantage of image self similarity. The method can be easily generalized to deal with other tasks, as for example stereo upsampling. The method is compared with state-of-the-art stereo methods, showing competitive results on the Middlebury stereo database.

We propose a new video denoising algorithm combining state of the art image and video denoising algorithms. We extend the DDID [1] algorithm to video sequences and then combine it with the SPTWO [2] method. The experimentation illustrates how the new method keeps the best of each algorithm, being superior both visually and numerically to other state of the art techniques.

We present a novel aggregation method based on adaptive support weights for local stereo matching. In order to correctly match each block, the adaptive weight distribution should favor pixels sharing the same disparity. State of the art algorithms make this configuration depend only on spatial and color differences, identifying disparity discontinuities with color ones. Compared to these algorithms, we introduce a weight support depending on each tested disparity and not only on the image configuration around the reference pixel. For each tested disparity, we favor pixels in the block matching with smaller cost, which are supposed to be more likely to be correctly represented by this disparity. Besides, we use a multiscale strategy with invalidation criteria to reduce match ambiguity and computational time. Results on the Middlebury stereo benchmark show the performance improvement of the proposed approach in comparison with state of the art.

We propose a depth estimation method for light field images. Light field images can be considered as a collection of 2D images taken from different viewpoints arranged in a regular grid. We exploit this configuration and compute the disparity maps between specific pairs of views. This computation is carried out by a state of the art two-view stereo method providing a non dense disparity estimation. We propose a disparity interpolation method increasing the density and improving the accuracy of this initial estimate. Disparities obtained from several pairs of views are fused to obtain a unique and robust estimation. Finally, different experiments on synthetic and real images show how the proposed method outperforms state of the art results.

Common satellite imagery products consist of a panchromatic image at high spatial resolution and several misregistered spectral bands at lower resolution. Pansharpening is the fusion process by which a high-resolution multispectral image is inferred. We propose a variational model for which pan-sharpening is defined as an optimization problem minimizing a cost function with nonlocal regularization. We incorporate a new term preserving the radiometric ratio between the panchromatic and each spectral band. The resulting model is channel-decoupled, thus permitting the application to misregistered spectral data. The experimental results illustrate the superiority of the proposed method to preserve spatial details, reduce color artifacts, and avoid aliasing.

In this work we propose a depth estimation method for light field images. Light field images can be considered as a collection of 2D images taken from different viewpoints arranged in a regular grid. This means that disparity is the same for any pair of consecutive views, in both vertical and horizontal directions. We exploit this fact by computing disparity maps between specific pairs of views. This computation is done by a state of the art two-view stereo method which also provides a mask of incorrect matches. Then, disparity maps and masks are used to obtain a unique and robust estimation. Finally, we show the performance of the algorithm by means of different experiments.

A novel image sequence denoising algorithm is presented. The proposed approach takes advantage of the self-similarity and redundancy of adjacent frames. The algorithm is inspired by fusion algorithms, and as the number of frames increases, it tends to a pure temporal average. The use of motion compensation by regularized optical flow methods permits robust patch comparison in a spatiotemporal volume. The use of principal component analysis ensures the correct preservation of fine texture and details. An extensive comparison with the state-of-the-art methods illustrates the superior performance of the proposed approach, with improved texture and detail reconstruction.

We consider the two-images stereo disparity problem favoring correctness of matches over density. We will deal with high resolution images which permit an accurate matching in textured zones, but which might present, as any stereo pair, ambiguities and occlusions. Global variational methods can estimate a dense map based on regularity assumptions about the disparity function. However, if these assumptions are incorrect this may lead to erroneous interpretations and mismatches. The availability of tristereo or even multiview stereo imagery permit us to combine the disparities from different pairs, allowing for a reliable densification not based on regularity assumptions. Local methods are suitable for this purpose since they permit us to check the validity of each match. The main disadvantages of local methods are the matching ambiguity and the failures of the fronto-parallel hypothesis (at places like discontinuities and slanted surfaces). We advocate, in this work, for the use of oriented windows in order to deal with slanted surfaces and discontinuities. Unlike adaptive support windows the oriented windows permit us to correctly estimate disparities on non-fronto-parallel surfaces. Several parameterless techniques for detecting mismatches are presented. The incorporation of these validation techniques in a coarse-to-fine multiwindow algorithm, allows us to obtain fairly dense results with few mismatches. An extensive comparison, including classical stereo pairs, high resolution satellite images, and images from the KITTI benchmark, illustrates the performance of the proposed method.

The camera calibration parameters and the image processing chain which generated a given image are generally not available to the receiver. This happens for example with scanned photographs and for most JPEG images. These images have undergone various nonlinear contrast changes and also linear and nonlinear filters. To deal with remnant noise in such images, we introduce a general nonparametric intensity and frequency-dependent noise model. We demonstrate by simulated and experiments with real images that this model, which requires the estimation of more than 1000 parameters, performs an efficient noise estimation. The proposed noise model is a patch model. Its estimation can therefore be used as a preliminary step to any patch-based denoising method. Our noise estimation method introduces several new tools for performing this complex estimation. One of them is a new sparse patch distance function permitting to find noisy patches with similar underlying geometry. A validation of the noise model and of its estimation method is obtained by comparing its results to ground-truth noise curves for both raw and JPEG-encoded images, and by visual inspection of the denoising results of real images. A fair comparison with the state of the art is also performed.

We present a non-parametric method estimating an intensity and frequency dependent noise from a single image. The noise model is estimated on image patches and can be used consequently in all patch-based denoising methods. The method applies to cases where no access is granted to the image noise model, in particular to scanned photographs and JPEG images. The general noise model and the method to evaluate it are validated by comparing the estimations with the corresponding ground-truth curves for raw and JPEG images. Denoising experiments on scanned photographs also support the efficiency of the estimation method.

Pansharpening refers to the fusion process of inferring a high-resolution multispectral image from a high-resolution panchromatic image and a low-resolution multispectral one. In this paper we propose a new variational method for pansharpening which incorporates a nonlocal regularization term and two fidelity terms, one describing the relation between the panchromatic image and the high-resolution spectral channels and the other one preserving the colors from the low-resolution modality. The nonlocal term is based on the image self-similarity principle applied to the panchromatic image. The existence and uniqueness of minimizer for the described functional is proved in a suitable space of weighted integrable functions. Although quite successful in terms of relative error, state-of-the-art pansharpening methods introduce relevant color artifacts. These spectral distortions can be significantly reduced by involving the image self-similarity. Extensive comparisons with state-of-the-art algorithms are performed.

Most common cameras use a CCD sensor device measuring a single color per pixel. The other two color values of each pixel must be interpolated from the neighboring pixels in the so-called demosaicking process. State-of-the-art demosaicking algorithms take advantage of inter-channel correlation locally selecting the best interpolation direction. These methods give impressive results except when local geometry cannot be inferred from neighboring pixels or channel correlation is low. In these cases, they create interpolation artifacts. We introduce a new algorithm involving non-local image self-similarity in order to reduce interpolation artifacts when local geometry is ambiguous. The proposed algorithm introduces a clear and intuitive manner of balancing how much channel-correlation must be taken advantage of. Comparison shows that the proposed algorithm gives state-of-the-art methods in several image bases.

This article proposes a new multiscale filter accelerating Monte Carlo renderer. Each pixel in the image is characterized by the colors of the rays that reach its surface. The proposed filter uses a statistical distance to compare with each other the ray color distributions associated with different pixels, at each scale. Based on this distance, it decides whether two pixels can share their rays or not. This simple and easily reproducible algorithm provides apsnrgain of 10 to 15 decibels, or equivalently accelerates the rendering process by using 10 to 30 times fewer samples without observable bias. The algorithm is consistent, does not assume a particular noise model, and is immediately extendable to synthetic movies. Being based on the ray color values only, it can be combined with all rendering effects.

Recent state-of-the-art image denoising methods use nonparametric estimation processes for $8 \times 8$ patches and obtain surprisingly good denoising results. The mathematical and experimental evidence of two recent articles suggests that we might even be close to the best attainable performance in image denoising ever. This suspicion is supported by a remarkable convergence of all analyzed methods. Still more interestingly, most patch-based image denoising methods can be summarized in one paradigm, which unites the transform thresholding method and a Markovian Bayesian estimation. As the present paper shows, this unification is complete when the patch space is assumed to be a Gaussian mixture. Each Gaussian distribution is associated with its orthonormal basis of patch eigenvectors. Thus, transform thresholding (or a Wiener filter) is made on these local orthogonal bases. In this paper a simple patch-based Bayesian method is proposed, which on the one hand keeps most interesting features of former methods, and on the other hand slightly improves the state of the art of color images.

This paper addresses the problem of obtaining a quality photograph of a painting by multi-image fusion methods. The problem is particularly challenging because of the uncontrolled illumination conditions and of the destructive reflection speckle present in most photographs of paintings. A fully automatic image processing chain is described that, starting from several bursts of a painting taken under different angles, permits one to obtain the best possible result by eliminating highlights and motion blur by robust statistics, reducing noise by fusion, and compensating optical distortion in the registration process. This image fusion method is applicable to photographs of a painting taken with a hand-held camera without any particular setup. It works under bad lighting conditions and eliminates motion blur, even when the painting is protected by a glass screen creating structured reflections of the room. The careful discussion of each step of the processing chain also permits one to review and discuss the efficiency of the image fusion tools recently proposed in the literature and insert several new ones in the chain.

We present a new method to automatically detect, track, sort, search, display and record meaningful changes in a video sequence. The meaningfulness of changes is measured in terms of their False Alarm Rate, which is controlled by the user. The process entails four main steps: a) background extraction; b) change detection; c) change tracking and recording; and d) classification of change according to its attributes (size, color, etc.). The information obtained from the classification step shall be used to index the video data.

Blotches are very common, localized, and non persistent impairments in digitized film archive. Many methods have been proposed so far for detecting them and restoring the underlying regions. Most detection techniques rely on the hypothesis that blotches contradict a model of motion regularity and, up to a prior motion compensation, correspond to significant temporal variations of intensity with respect to a global threshold. In this paper, we propose a statistical approach to detect blotches in image sequences, which yields thresholds adapted to the local statistics of the frames, and which takes into account gray level differences in neighborhoods instead of isolated points. This approach is combined with a block-based motion estimation. The whole procedure is confronted with classical approaches on several sequences.

Can images be decomposed into the sum of a geometric part and a textural part? In a theoretical breakthrough, [Y. Meyer, Oscillating Patterns in Image Processing and Nonlinear Evolution Equations. Providence, RI: American Mathematical Society, 2001] proposed variational models that force the geometric part into the space of functions with bounded variation, and the textural part into a space of oscillatory distributions. Meyer's models are simple minimization problems extending the famous total variation model. However, their numerical solution has proved challenging. It is the object of a literature rich in variants and numerical attempts. This paper starts with the linear model, which reduces to a low-pass/high-pass filter pair. A simple conversion of the linear filter pair into a nonlinear filter pair involving the total variation is introduced. This new-proposed nonlinear filter pair retains both the essential features of Meyer's models and the simplicity and rapidity of the linear model. It depends upon only one transparent parameter: the texture scale, measured in pixel mesh. Comparative experiments show a better and faster separation of cartoon from texture. One application is illustrated: edge detection.

Demosaicking is the process by which from a matrix of colored pixels measuring only one color component per pixel, red, green, or blue, one can infer a whole color information at each pixel. This inference requires a deep understanding of the interaction between colors, and the involvement of image local geometry. Although quite successful in making such inferences with very small relative error, state-of-the-art demosaicking methods fail when the local geometry cannot be inferred from the neighboring pixels. In such a case, which occurs when thin structures or fine periodic patterns were present in the original, state-of-the-art methods can create disturbing artifacts, known as zipper effect, blur, and color spots. The aim of this paper is to show that these artifacts can be avoided by involving the image self-similarity to infer missing colors. Detailed experiments show that a satisfactory solution can be found, even for the most critical cases. Extensive comparisons with state-of-the-art algorithms will be performed on two different classic image databases.

One of the central problems of medical imaging is the three-dimensional (3D) visualization of body parts. The 3D volume can be viewed in slices, but the extraction of a part requires a segmentation process. Inasmuch as body parts are distinguishable by their various densities, a widely accepted method for extracting an organ is to extract isodensity surfaces by a simple threshold. Unfortunately, the density of organs, arteries, etc. varies spatially due to morphology, and no unique threshold allows one to extract the organs boundaries. The snake or active contour methods have attempted to capture these boundaries as smooth and overall contrasted surfaces. The snake method suffers, however, from severe drawbacks. The contour has to be initialized near the boundary. In addition, many body parts have too complex a topology. In this paper we focus on another idea, which is to preprocess the image before thresholding. The preprocessing aims at the homogeneity of the different parts while preserving small features. Starting from a recent seminal work by Grady and Funka-Lea [in Computer Vision and Mathematical Methods in Medical and Biomedical Image Analysis: ECCV 2004 Workshops CVAMIA and MMBIA, Prague, Czech Republic, May 2004, Revised Selected Papers, Springer, Berlin, 2004, pp. 230–245], several linear heat equations on images will be compared. They stem from nonlinear partial differential equations or from their associated nonlinear filters. By linearizing these processes one obtains more accurate topology preserving methods. These linear filters will be tested comparatively to visualize challenging angiography images of arteries. A salient fact of the method will emerge. By a concentration phenomenon, peaks in the image histogram become much more concentrated under the linear heat equations, thus permitting us to fix the thresholds defining the surfaces without supervision. Automatic extraction can be performed in this way for angiography images taken at a one-year or longer delay.

Many classical image denoising methods are based on a local averaging of the color, which increases the signal/noise ratio. One of the most used algorithms is the neighborhood filter by Yaroslavsky or sigma filter by Lee, also called in a variant "SUSAN" by Smith and Brady or "Bilateral filter" by Tomasi and Manduchi. These filters replace the actual value of the color at a point by an average of all values of points which are simultaneously close in space and in color. Unfortunately, these filters show a "staircase effect," that is, the creation in the image of flat regions separated by artifact boundaries. In this paper, we first explain the staircase effect by finding the subjacent partial differntial equation (PDE) of the filter. We show that this ill-posed PDE is a variant of another famous image processing model, the Perona-Malik equation, which suffers the same artifacts. As we prove, a simple variant of the neighborhood filter solves the problem. We find the subjacent stable PDE of this variant. Finally, we apply the same correction to the recently introduced NL-means algorithm which had the same staircase effect, for the same reason.

State of the art movie restoration methods either estimate motion and filter out the trajectories, or compensate the motion by an optical flow estimate and then filter out the compensated movie. Now, the motion estimation problem is ill posed. This fact is known as the aperture problem: trajectories are ambiguous since they could coincide with any promenade in the space-time isophote surface. In this paper, we try to show that, for denoising, the aperture problem can be taken advantage of. Indeed, by the aperture problem, many pixels in the neighboring frames are similar to the current pixel one wishes to denoise. Thus, denoising by an averaging process can use many more pixels than just the ones on a single trajectory. This observation leads to use for movies a recently introduced image denoising method, the NL-means algorithm. This static 3D algorithm outperforms motion compensated algorithms, as it does not lose movie details. It involves the whole movie isophote and not just a trajectory.

We propose a new measure, the method noise, to evaluate and compare the performance of digital image denoising methods. We first compute and analyze this method noise for a wide class of denoising algorithms, namely the local smoothing filters. Second, we propose a new algorithm, the nonlocal means (NL-means), based on a nonlocal averaging of all pixels in the image. Finally, we present some experiments comparing the NL-means algorithm and the local smoothing filters.

In this work, we present and analyze an image denoising method, the NL-means algorithm, based on a non local averaging of all pixels in the image. We also introduce the concept of method noise, that is, the difference between the original (always slightly noisy) digital image and its denoised version. Finally, we present some experiences comparing the NL-means results with some classical denoising methods.