Text-to-image latent diffusion models (LDMs) have recently emerged as powerful generative models with great potential for solving inverse problems in imaging. However, leveraging such models in a Plug & Play (PnP), zero-shot manner remains challenging because it requires identifying a suitable text prompt for the unknown image of interest. Also, existing text-to-image PnP approaches are highly computationally expensive. We herein address these challenges by proposing a novel PnP inference paradigm specifically designed for embedding generative models within stochastic inverse solvers, with special attention to Latent Consistency Models (LCMs), which distill LDMs into fast generators. We leverage our framework to propose LAtent consisTency INverse sOlver (LATINO), the first zero-shot PnP framework to solve inverse problems with priors encoded by LCMs. Our conditioning mechanism avoids automatic differentiation and reaches SOTA quality in as little as 8 neural function evaluations. As a result, LATINO delivers remarkably accurate solutions and is significantly more memory and computationally efficient than previous approaches. We then embed LATINO within an empirical Bayesian framework that automatically calibrates the text prompt from the observed measurements by marginal maximum likelihood estimation. Extensive experiments show that prompt self-calibration greatly improves estimation, allowing LATINO with PRompt Optimization to define new SOTAs in image reconstruction quality and computational efficiency. The code is available at https://latino-pro.github.io

We consider the problem of sampling distributions stemming from non-convex potentials with Unadjusted Langevin Algorithm (ULA). We prove the stability of the discrete-time ULA to drift approximations under the assumption that the potential is strongly convex at infinity. In many context, e.g. imaging inverse problems, potentials are non-convex and non-smooth. Proximal Stochastic Gradient Langevin Algorithm (PSGLA) is a popular algorithm to handle such potentials. It combines the forward-backward optimization algorithm with a ULA step. Our main stability result combined with properties of the Moreau envelope allows us to derive the first proof of convergence of the PSGLA for non-convex potentials. We empirically validate our methodology on synthetic data and in the context of imaging inverse problems. In particular, we observe that PSGLA exhibits faster convergence rates than Stochastic Gradient Langevin Algorithm for posterior sampling while preserving its restoration properties.

Plug-and-Play (PnP) algorithms are a class of iterative algorithms that address image inverse problems by combining a physical model and a deep neural network for regularization. Even if they produce impressive image restoration results, these algorithms rely on a non-standard use of a denoiser on images that are less and less noisy along the iterations, which contrasts with recent algorithms based on Diffusion Models (DM), where the denoiser is applied only on re-noised images. We propose a new PnP framework, called Stochastic deNOising REgularization (SNORE), which applies the denoiser only on images with noise of the adequate level. It is based on an explicit stochastic regularization, which leads to a stochastic gradient descent algorithm to solve ill-posed inverse problems. A convergence analysis of this algorithm and its annealing extension is provided. Experimentally, we prove that SNORE is competitive with respect to state-of-the-art methods on deblurring and inpainting tasks, both quantitatively and qualitatively.

Shadows in videos are difficult to detect because of the large shadow deformation between frames. In this work, we argue that accounting for shadow deformation is essential when designing a video shadow detection method. To this end, we introduce the shadow deformation attention trajectory (SODA), a new type of video self-attention module, specially designed to handle the large shadow deformations in videos. Moreover, we present a new shadow contrastive learning mechanism (SCOTCH) which aims at guiding the network to learn a unified shadow representation from massive positive shadow pairs across different videos. We demonstrate empirically the effectiveness of our two contributions in an ablation study. Furthermore, we show that SCOTCH and SODA significantly outperforms existing techniques for video shadow detection. Code is available at the project page: https://lihaoliu-cambridge.github.io/scotch_and_soda/

In this paper, we propose to regularize ill-posed inverse problems using a deep hierarchical variational autoencoder (HVAE) as an image prior. The proposed method synthesizes the advantages of i) denoiser-based Plug & Play approaches and ii) generative model based approaches to inverse problems. First, we exploit VAE properties to design an efficient algorithm that benefits from convergence guarantees of Plug-and-Play (PnP) methods. Second, our approach is not restricted to specialized datasets and the proposed PnP-HVAE model is able to solve image restoration problems on natural images of any size. Our experiments show that the proposed PnP-HVAE method is competitive with both SOTA denoiser-based PnP approaches, and other SOTA restoration methods based on generative models. The code for this project is available at https://github.com/jprost76/PnP-HVAE.

Plug-and-Play (PnP) methods are efficient iterative algorithms for solving ill-posed image inverse problems. PnP methods are obtained by using deep Gaussian denoisers instead of the proximal operator or the gradient-descent step within proximal algorithms. Current PnP schemes rely on data-fidelity terms that have either Lipschitz gradients or closed-form proximal operators, which is not applicable to Poisson inverse problems. Based on the observation that the Gaussian noise is not the adequate noise model in this setting, we propose to generalize PnP using the Bregman Proximal Gradient (BPG) method. BPG replaces the Euclidean distance with a Bregman divergence that can better capture the smoothness properties of the problem. We introduce the Bregman Score Denoiser specifically parametrized and trained for the new Bregman geometry and prove that it corresponds to the proximal operator of a nonconvex potential. We propose two PnP algorithms based on the Bregman Score Denoiser for solving Poisson inverse problems. Extending the convergence results of BPG in the nonconvex settings, we show that the proposed methods converge, targeting stationary points of an explicit global functional. Experimental evaluations conducted on various Poisson inverse problems validate the convergence results and showcase effective restoration performance.

Plug-and-Play (PnP) methods solve ill-posed inverse problems through iterative proximal algorithms by replacing a proximal operator by a denoising operation. When applied with deep neural network denoisers, these methods have shown state-of-the-art visual performance for image restoration problems. However, their theoretical convergence analysis is still incomplete. Most of the existing convergence results consider nonexpansive denoisers, which is non-realistic, or limit their analysis to strongly convex data-fidelity terms in the inverse problem to solve. Recently, it was proposed to train the denoiser as a gradient descent step on a functional parameterized by a deep neural network. Using such a denoiser guarantees the convergence of the PnP version of the Half-Quadratic-Splitting (PnP-HQS) iterative algorithm. In this paper, we show that this gradient denoiser can actually correspond to the proximal operator of another scalar function. Given this new result, we exploit the convergence theory of proximal algorithms in the nonconvex setting to obtain convergence results for PnP-PGD (Proximal Gradient Descent) and PnP-ADMM (Alternating Direction Method of Multipliers). When built on top of a smooth gradient denoiser, we show that PnP-PGD and PnP-ADMM are convergent and target stationary points of an explicit functional. These convergence results are confirmed with numerical experiments on deblurring, super-resolution and inpainting.

Neural networks have revolutionized the field of data science, yielding remarkable solutions in a data-driven manner. For instance, in the field of mathematical imaging, they have surpassed traditional methods based on convex regularization. However, a fundamental theory supporting the practical applications is still in the early stages of development. We take a fresh look at neural networks and examine them via nonlinear eigenvalue analysis. The field of nonlinear spectral theory is still emerging, providing insights about nonlinear operators and systems. In this paper we view a neural network as a complex nonlinear operator and attempt to find its nonlinear eigenvectors. We first discuss the existence of such eigenvectors and analyze the kernel of ReLU networks. Then we study a nonlinear power method for generic nonlinear operators. For proximal operators associated to absolutely one-homogeneous convex regularization functionals, we can prove convergence of the method to an eigenvector of the proximal operator. This motivates us to apply a nonlinear method to networks which are trained to act similarly as a proximal operator. In order to take the non-homogeneity of neural networks into account we define a modified version of the power method. We perform extensive experiments for different proximal operators and on various shallow and deep neural networks designed for image denoising. Proximal eigenvectors will be used for geometric analysis of graphs, as clustering or the computation of distance functions. For simple neural nets, we observe the influence of training data on the eigenvectors. For state-of-the-art denoising networks, we show that eigenvectors can be interpreted as (un)stable modes of the network, when contaminated with noise or other degradations.

Fully supervised deep neural networks for segmentation usually require a massive amount of pixel-level labels which are manually expensive to create. In this work, we develop a multi-task learning method to relax this constraint. We regard the segmentation problem as a sequence of approximation subproblems that are recursively defined and in increasing levels of approximation accuracy. The subproblems are handled by a framework that consists of 1) a segmentation task that learns from pixel-level ground truth segmentation masks of a small fraction of the images, 2) a recursive approximation task that conducts partial object regions learning and data-driven mask evolution starting from partial masks of each object instance, and 3) other problem oriented auxiliary tasks that are trained with sparse annotations and promote the learning of dedicated features. Most training images are only labeled by (rough) partial masks, which do not contain exact object boundaries, rather than by their full segmentation masks. During the training phase, the approximation task learns the statistics of these partial masks, and the partial regions are recursively increased towards object boundaries aided by the learned information from the segmentation task in a fully data-driven fashion. The network is trained on an extremely small amount of precisely segmented images and a large set of coarse labels. Annotations can thus be obtained in a cheap way. We demonstrate the efficiency of our approach in three applications with microscopy images and ultrasound images.

We propose a new variational model for non-linear image fusion. Our approach is based on the use of an osmosis energy term related to the one studied in Vogel et al. [44] and Weickert et al. [45]. The minimization of the proposed non-convex energy realizes visually plausible image data fusion, invariant to multiplicative brightness changes. On the practical side, it requires minimal supervision and parameter tuning and can encode prior information on the structure of the images to be fused. For the numerical solution of the proposed model, we develop a primal-dual algorithm and we apply the resulting minimization scheme to solve multi-modal face fusion, color transfer and cultural heritage conservation problems. Visual and quantitative comparisons to state-of-the-art approaches prove the out-performance and the flexibility of our method.

Most superpixel algorithms compute a trade-off between spatial and color features at the pixel level. Hence, they may need fine parameter tuning to balance the two measures, and highly fail to group pixels with similar local texture properties. In this paper, we address these issues with a new Texture-Aware SuperPixel (TASP) method. To accurately segment textured and smooth areas, TASP automatically adjusts its spatial constraint according to the local feature variance. Then, to ensure texture homogeneity within superpixels, a new pixel to super-pixel patch-based distance is proposed. TASP outperforms the segmentation accuracy of the state-of-the-art methods on texture and also natural color image datasets.

In this paper, we present a graph-based semi-supervised framework for hyperspectral image classification. We first introduce a novel superpixel algorithm based on the spectral covariance matrix representation of pixels to provide a better representation of our data. We then construct a superpixel graph, based on carefully considered feature vectors, before performing classification. We demonstrate, through a set of experimental results using two benchmarking datasets, that our approach outperforms three state-of-the-art classification frameworks, especially when a extremely small amount of labelled data is used.

This paper presents a unified framework for smooth convex regularization of discrete optimal transport problems. In this context, the regularized optimal transport turns out to be equivalent to a matrix nearness problem with respect to Bregman divergences. Our framework thus naturally generalizes a previously proposed regularization based on the Boltzmann-Shannon entropy related to the Kullback-Leibler divergence, and solved with the Sinkhorn-Knopp algorithm. We call the regularized optimal transport distance the rot mover's distance in reference to the classical earth mover's distance. By exploiting alternate Bregman projections, we develop the alternate scaling algorithm and non-negative alternate scaling algorithm, to compute efficiently the regularized optimal plans depending on whether the domain of the regularizer lies within the non-negative orthant or not. We further enhance the separable case with a sparse extension to deal with high data dimensions. We also instantiate our framework and discuss the inherent specificities for well-known regularizers and statistical divergences in the machine learning and information geometry communities. Finally, we demonstrate the merits of our methods with experiments using synthetic data to illustrate the effect of different regularizers, penalties and dimensions, as well as real-world data for a pattern recognition application to audio scene classification.

Nonlinear eigenfunctions, induced by subgradients of one-homogeneous functionals (such as the 1-Laplacian), have shown to be instrumental in segmentation, clustering, and image decomposition. We present a class of flows for finding such eigenfunctions, generalizing a method recently suggested by Nossek and Gilboa. We analyze the flows on grids and graphs in the time-continuous and time-discrete settings. For a specific type of flow within this class, we prove convergence of the numerical iterations procedure and prove existence and uniqueness of the time-continuous case. Several toy examples are provided for illustrating the theoretical results, showing how such flows can be used on images and graphs.

In this work, we propose a fast superpixel-based color transfer method (SCT) between two images. Superpixels enable to decrease the image dimension and to extract a reduced set of color candidates. We propose to use a fast approximate nearest neighbor matching algorithm in which we enforce the match diversity by limiting the selection of the same superpixels. A fusion framework is designed to transfer the matched colors, and we demonstrate the improvement obtained over exact matching results. Finally, we show that SCT is visually competitive compared to state-of-the-art methods.

Regular decompositions are necessary for most superpixel-based object recognition or tracking applications. So far in the literature, the regularity or compactness of a superpixel shape is mainly measured by its circularity. In this work, we first demonstrate that such measure is not adapted for super-pixel evaluation, since it does not directly express regularity but circular appearance. Then, we propose a new metric that considers several shape regularity aspects: convexity, balanced repartition, and contour smoothness. Finally, we demonstrate that our measure is robust to scale and noise and enables to more relevantly compare superpixel methods.

In this paper, we propose a new framework to remove parts of the systematic errors affecting popular restoration algorithms, with a special focus on image processing tasks. Generalizing ideas that emerged for $\ell_1$ regularization, we develop an approach refitting the results of standard methods toward the input data. Total variation regularizations and nonlocal means are special cases of interest. We identify important covariant information that should be preserved by the refitting method and emphasize the importance of preserving the Jacobian (w.r.t. the observed signal) of the original estimator. Then, we provide an approach that has a “twicing” flavor and allows refitting the restored signal by adding back a local affine transformation of the residual term. We illustrate the benefits of our method on numerical simulations for image restoration tasks.

Superpixels have become very popular in many computer vision applications. Nevertheless, they remain under-exploited, since the superpixel decomposition may produce irregular and nonstable segmentation results due to the dependency to the image content. In this paper, we first introduce a novel structure, a superpixel-based patch, called SuperPatch. The proposed structure, based on superpixel neighborhood, leads to a robust descriptor, since spatial information is naturally included. The generalization of the PatchMatch method to SuperPatches, named SuperPatchMatch, is introduced. Finally, we propose a framework to perform fast segmentation and labeling from an image database, and demonstrate the potential of our approach, since we outperform, in terms of computational cost and accuracy, the results of state-of-the-art methods on both face labeling and medical image segmentation.

Superpixel decomposition methods are generally used as a pre-processing step to speed up image processing tasks. They group the pixels of an image into homogeneous regions while trying to respect existing contours. For all state-of-the-art superpixel decomposition methods, a trade-off is made between 1) computational time, 2) adherence to image contours and 3) regularity and compactness of the decomposition. In this paper, we propose a fast method to compute Superpixels with Contour Adherence using Linear Path (SCALP) in an iterative clustering framework. The distance computed when trying to associate a pixel to a superpixel during the clustering is enhanced by considering the linear path to the superpixel barycenter. The proposed framework produces regular and compact superpixels that adhere to the image contours. We provide a detailed evaluation of SCALP on the standard Berkeley Segmentation Dataset. The obtained results outperform state-of-the-art methods in terms of standard superpixel and contour detection metrics.

This paper provides a new method to colorize gray-scale images. While the computation of the luminance channel is directly performed by a linear transformation, the colorization process is an ill-posed problem that requires some priors. In the literature two classes of approach exist. The first class includes manual methods that need the user to manually add colors on the image to colorize. The second class includes exemplar-based approaches where a color image, with a similar semantic content, is provided as input to the method. These two types of priors have their own advantages and drawbacks. In this paper, a new variational framework for exemplar-based colorization is proposed. A nonlocal approach is used to find relevant color in the source image in order to suggest colors on the gray-scale image. The spatial coherency of the result as well as the final color selection is provided by a nonconvex variational framework based on a total variation. An efficient primal-dual algorithm is provided, and a proof of its convergence is proposed. In this work, we also extend the proposed exemplar-based approach to combine both exemplar-based and manual methods. It provides a single framework that unifies advantages of both approaches. Finally, experiments and comparisons with state-of-the-art methods illustrate the efficiency of our proposal.

This paper deals with the problem of image colorization. A model including total variation regularization is proposed. Our approach colorizes directly the three RGB channels, while most existing methods were only focusing on the two chrominance channels. By using the three channels, our approach is able to better preserve color consistency. Our model is non convex, but we propose an efficient primal-dual like algorithm to compute a local minimizer. Numerical examples illustrate the good behavior of our algorithm with respect to state-of-the-art methods.

This paper studies the problem of color transfer between images using optimal transport techniques. While being a generic framework to handle statistics properly, it is also known to be sensitive to noise and outliers, and is not suitable for direct application to images without additional postprocessing regularization to remove artifacts. To tackle these issues, we propose to directly deal with the regularity of the transport map and the spatial consistency of the reconstruction. Our approach is based on the relaxed and regularized discrete optimal transport method of [1]. We extend this work by (i) modeling the spatial distribution of colors within the image domain and (ii) tuning automatically the relaxation parameters. Experiments on real images demonstrate the capacity of our model to adapt itself to the considered data.

This article introduces a generalization of the discrete optimal transport, with applications to color image manipulations. This new formulation includes a relaxation of the mass conservation constraint and a regularization term. These two features are crucial for image processing tasks where one must take into account families of multimodal histograms with large mass variation across modes. The corresponding relaxed and regularized transportation problem is the solution of a convex optimization problem. Depending on the regularization used, this minimization can be solved using standard linear programming methods or first order proximal splitting schemes. The resulting transportation plan can be used as a color transfer map, which is robust to mass variation across image color palettes. Furthermore, the regularization of the transport plan helps remove colorization artifacts due to noise amplification. We also extend this framework to compute the barycenter of distributions. The barycenter is the solution of an optimization problem, which is separately convex with respect to the barycenter and the transportation plans, but not jointly convex. A block coordinate descent scheme converges to a stationary point of the energy. We show that the resulting algorithm can be used for color normalization across several images. The relaxed and regularized barycenter defines a common color palette for those images. Applying color transfer toward this average palette performs a color normalization of the input images.

This article reviews the use of first order convex optimization schemes to solve the discretized dynamic optimal transport problem, initially proposed by Benamou and Brenier. We develop a staggered grid discretization that is well adapted to the computation of the $L^2$ optimal transport geodesic between distributions defined on a uniform spatial grid. We show how proximal splitting schemes can be used to solve the resulting large scale convex optimization problem. A specific instantiation of this method on a centered grid corresponds to the initial algorithm developed by Benamou and Brenier. We also show how more general cost functions can be taken into account and how to extend the method to perform optimal transport on a Riemannian manifold.

In this paper, we address the problem of recovering a color image from a grayscale one. The input color data comes from a source image considered as a reference image. Reconstructing the missing color of a grayscale pixel is here viewed as the problem of automatically selecting the best color among a set of color candidates while simultaneously ensuring the local spatial coherency of the reconstructed color information. To solve this problem, we propose a variational approach where a specific energy is designed to model the color selection and the spatial constraint problems simultaneously. The contributions of this paper are twofold. First, we introduce a variational formulation modeling the color selection problem under spatial constraints and propose a minimization scheme, which computes a local minima of the defined nonconvex energy. Second, we combine different patch-based features and distances in order to construct a consistent set of possible color candidates. This set is used as input data and our energy minimization automatically selectsthe best color to transfer for each pixel of the grayscale image. Finally, the experiments illustrate the potentiality of our simple methodology and show that our results are very competitive with respect to the state-of-the-art methods.

This paper is concerned with the relaxation of nonconvex functionals used in image processing. We review most of the recently introduced relaxation methods, and we propose a new convex one based on a probabilistic approach, which has the advantages of being intuitive, flexible, and involving an algorithm without inner loops. We investigate in detail the connections between the solutions of the relaxed functionals with minimizers of the original one. Such connection is demonstrated only for a nonconvex relaxation which turns out to be quite robust to initialization. As a case of study, we illustrate our theoretical analysis with numerical experiments, namely, for the optical flow problem.

This paper deals with the problem of segmenting an image with active contours. We explain how recent convexification methods allow now to use active contours without level sets with simple and efficient first order schemes. We recall different algorithms proposed in the literature, and we propose a new variant. Numerical experiments in 2D and 3D confirm the interest of the approach.

In this paper we propose an algorithm for stereoscopic image inpainting, given the inpainting mask in both images. We also assume that depth map is known in one of the images of the stereo pair, taken as reference. This image is clustered in homogeneous color regions using a mean-shift procedure. In each clustered region, depths are fitted by planes and then extended into the mask. Then we inpaint the visible parts of each extended region using a modified exemplar-based inpainting algorithm. Finally, we extend the algorithm to stereoscopic image inpainting. We display some experiments showing the performance of the proposed algorithm.

This work presents a new method for tracking and segmenting along time-interacting objects within an image sequence. One major contribution of the paper is the formalization of the notion of visible and occluded parts. For each object, we aim at tracking these two parts. Assuming that the velocity of each object is driven by a dynamical law, predictions can be used to guide the successive estimations. Separating these predicted areas into good and bad parts with respect to the final segmentation and representing the objects with their visible and occluded parts permit handling partial and complete occlusions. To achieve this tracking, a label is assigned to each object and an energy function representing the multilabel problem is minimized via a graph cuts optimization. This energy contains terms based on image intensities which enable segmenting and regularizing the visible parts of the objects. It also includes terms dedicated to the management of the occluded and disappearing areas, which are defined on the areas of prediction of the objects. The results on several challenging sequences prove the strength of the proposed approach.

In this paper, we propose a variational formulation for histogram transfer of two or more color images. We study an energy functional composed by three terms: one tends to approach the cumulative histograms of the transformed images, the other two tend to maintain the colors and geometry of the original images. By minimizing this energy, we obtain an algorithm that balances equalization and the conservation of features of the original images. As a result, they evolve while approaching an intermediate histogram between them. This intermediate histogram does not need to be specified in advance, but it is a natural result of the model. Finally, we provide experiments showing that the proposed method compares well with the state of the art.

In this paper the problem of optical flow and occlusion mask estimation is aborded. To that end, we consider a multi-label representation of the optical flow and we define an energy that models the problem. The convexification of the energy and its minimization with an iterative algorithm are studied. Our algorithm is implemented in GPU, since each pixel can be processed in parallel. From our experiments, the relation between the quality of the results obtained and computing time seems to be very promising.

In this paper we propose an algorithm for in painting of stereo images. The issue is to reconstruct the holes in a pair of stereo image as if they were the projection of a 3D scene. Hence, the reconstruction of the missing information has to produce a consistent visual perception of depth. Thus, first step of the algorithm consists in the computation and in painting of disparity maps in the given holes. The second step of the algorithm is to fill-in missing regions using the complete disparity maps in a way that avoids the creation of 3D artifacts. We present some experiments on several pairs of stereo images.

The complexity of dynamical laws governing 3D atmospheric flows associated with incomplete and noisy observations make the recovery of atmospheric dynamics from satellite images sequences very difficult. In this paper, we face the challenging problem of estimating physical sound and time-consistent horizontal motion fields at various atmospheric depths for a whole image sequence. Based on a vertical decomposition of the atmosphere, we propose a dynamically consistent atmospheric motion estimator relying on a multi-layer dynamical model. This estimator is based on a weak constraint variational data assimilation scheme and is applied on noisy and incomplete pressure difference observations derived from satellite images. The dynamical model consists in a simplified vorticity-divergence form of a multi-layer shallow-water model. Average horizontal motion fields are estimated for each layer. The performance of the proposed technique is assessed on real world meteorological satellite image sequences.

In this paper, a variational technique derived from optimal control theory is used in order to realize a dynamically consistent motion estimation of a whole fluid image sequence. The estimation is conducted through an iterative process involving a forward integration of a given dynamical model followed by a backward integration of an adjoint evolution law. By combining physical conservation laws and image observations, a physically grounded temporal consistency is imposed, and the quality of the motion estimation is significantly improved. The method is validated on two synthetic image sequences provided by numerical simulation of fluid flows and on real world meteorological examples.

In this paper, we present a framework for dynamic consistent estimation of dense motion fields over a sequence of images. The originality of the approach is to exploit recipes related to optimal control theory. This setup allows performing the estimation of an unknown state function according to a given dynamical model and to noisy and incomplete measurements. The overall process is formalized through the minimization of a global spatio-temporal cost functional w.r.t the complete sequence of motion fields. The minimization is handled considering an adjoint formulation. The resulting scheme consists in iterating a forward integration of the evolution model and a backward integration of the adjoint evolution model guided by a discrepancy measurement between the state variable and the available noisy observations. Such an approach allows us to cope with several delicate situations (such as the absence of data) which are not well managed with usual estimators.

In this paper, a new framework for the tracking of closed curves is described. The proposed approach, formalized through an optimal control technique, enables a continuous tracking along an image sequence of a deformable curve. The associated minimization process consists in a forward integration of a dynamical model followed by a backward integration of an adjoint dynamics. This latter pde includes a term related to the discrepancy between the state variables evolution law and discrete noisy measurements of the system. The closed curves are represented through an implicit surface.

In this paper, we present a framework for dynamic consistent estimation of dense motion fields over a sequence of Meteosat Second Generation (MSG) images. The originality of the approach is to exploit recipes related to optimal control theory developed in geophysical sciences. This framework, known as variational data assimilation, enables to perform the estimation of an unknown state function according to a given dynamical model and to noisy and incomplete measurements. In our work, the measurements are defined according to a smoothed brightness consistency model whereas the dynamical model on which we rely is derived from a velocity conservation law. The overall assimilation process is formalized through the minimization of a global spatio- temporal cost functional w.r.t to the complete sequence of motion fields. The minimization is handled considering an adjoint formulation. The resulting scheme consists in iterating a forward integration of the evolution model and a backward integration of the adjoint evolution model guided by a discrepancy measurement between the state variable and the available noisy observations. Such an approach allows us to cope with several delicate situations (such as the absence of data) which are not well managed with usual estimators. The efficiency of our approach is demonstrated on real data. It enables to estimate a sequence of dense motion fields even in situations where data are strongly corrupted.

In this paper, we address the problem of estimating mesoscale dynamics of atmospheric layers from satellite image sequences. Due to the great deal of spatial and temporal distortions of cloud patterns and because of the sparse 3-D nature of cloud observations, standard dense-motion field-estimation techniques used in computer vision are not well adapted to satellite images. Relying on a physically sound vertical decomposition of the atmosphere into layers, we propose a dense-motion estimator dedicated to the extraction of multilayer horizontal wind fields. This estimator is expressed as the minimization of a global function including data and spatio-temporal smoothness terms. A robust data term relying on the integrated-continuity equation mass-conservation model is proposed to fit sparse-transmittance observations related to each layer. A novel spatio-temporal smoother derived from large eddy prediction of a shallow-water momentum-conservation model is used to build constraints for large-scale temporal coherence. These constraints are combined in a global smoothing framework with a robust second-order smoother, preserving divergent and vorticity structures of the flow. For optimization, a two-stage motion estimation scheme is proposed to overcome multiresolution limitations when capturing the dynamics of mesoscale structures. This alternative approach relies on the combination of correlation and optical-flow observations in a variational context. An exhaustive evaluation of the novel method is first performed on a scalar image sequence generated by direct numerical simulation of a turbulent 2-D flow. By qualitative comparisons, the method is then assessed on a METEOSAT image sequence.