Rotoscoping, the detailed delineation of scene elements through a video shot, is a painstaking task of tremendous importance in professional post-production pipelines. While pixel-wise segmentation techniques can help for this task, professional rotoscoping tools rely on parametric curves that offer the artists a much better interactive control on the definition, editing and manipulation of the segments of interest. Sticking to this prevalent rotoscoping paradigm, we propose a novel framework to capture and track the visual aspect of an arbitrary object in a scene, given an initial closed outline of this object. This model combines a collection of local foreground/background appearance models spread along the outline, a global appearance model of the enclosed object and a set of distinctive foreground landmarks. The structure of this rich appearance model allows simple initialization, efficient iterative optimization with exact minimization at each step, and on-line adaptation in videos. We further extend this model by so-called trimaps which serve as an input to alpha-matting algorithms to allow truly seamless compositing. To this end, we leverage local classifiers attached to the roto-curves to define a confidence measure that is well-suited to define trimaps with adaptive band-widths. The resulting trimaps are parametric, temporally consistent and remain fully editable by the artist. We demonstrate qualitatively and quantitatively the merit of this framework through comparisons with tools based on either dynamic segmentation with a closed curve or pixel-wise binary labelling.

The problem of localizing occlusions between consecutive frames of a video is important but rarely tackled on its own. In most works, it is tightly interleaved with the computation of accurate optical flows, which leads to a delicate chicken-and-egg problem. With this in mind, we propose a novel approach to occlusion detection where visibility or not of a point in next frame is formulated in terms of visual reconstruction. The key issue is now to determine how well a pixel in the first image can be "reconstructed" from co-located colors in the next image. We first exploit this reasoning at the pixel level with a new detection criterion. Contrary to the ubiquitous displaced-framedifference and forward-backward flow vector matching, the proposed alternative does not critically depend on a precomputed, dense displacement field, while being shown to be more effective. We then leverage this local modeling within an energy-minimization framework that delivers occlusion maps. An easy-to-obtain collection of parametric motion models is exploited within the energy to provide the required level of motion information. Our approach outperforms state-of-the-art detection methods on the challenging MPI Sintel dataset.

A number of applications in video analysis rely on a per-frame motion segmentation of the scene as key preprocessing step. Moreover, different settings in video production require extracting segmentation masks of multiple moving objects and object parts in a hierarchical fashion. In order to tackle this problem, we propose to analyze and exploit the compositional structure of scene motion to provide a segmentation which is not purely driven by local image information. Specifically, we leverage a hierarchical motion-based partition of the scene to capture a mid-level understanding of the dynamic video content. We present experimental results showing the strengths of this approach in comparison to current video segmentation approaches.

We present a method to segment objects of interest in video sequences by combining robust background and foreground point tracking with joint color and motion-based segmentation. Our approach is sequential in time, avoiding a global processing of the video, while being simple and generic. This makes the method attractive for online applications, including video editing or augmented reality, as it can be adapted for both automated and interactive work-flows. We present visual and quantitative experiments to compare with existing algorithms, showing promising results.

We analyze the problem of how to correctly construct dense point trajectories from optical flow fields. First, we show that simple Euler integration is unavoidably inaccurate, no matter how good is the optical flow estimator. Then, an inverse integration scheme is analyzed which is more robust to bias and input noise and shows better stability properties. Our contribution is threefold: 1) a theoretical analysis that demonstrates why and in what sense inverse integration is more accurate; 2) a rich experimental validation both on synthetic and real (image) data; and 3) an algorithm for approximate online inverse integration. This new technique is precious whether one is trying to propagate information densely available on a reference frame to the other frames in the sequence or, conversely, to assign information densely over each frame by pulling it from the reference.

Accurate estimation of dense point correspondences between two distant frames of a video sequence is a challenging task. To address this problem, we present a combinatorial multistep integration procedure which allows one to obtain a large set of candidate motion fields between the two distant frames by considering multiple motion paths across the video sequence. Given this large candidate set, we propose to perform the optimal motion vector selection by combining a global optimization stage with a new statistical processing. Instead of considering a selection only based on intrinsic motion field quality and spatial regularization, the statistical processing exploits the spatial distribution of candidates and introduces an intra-candidate quality based on forward-backward consistency. Experiments evaluate the effectiveness of our method for distant motion estimation in the context of video editing.

A motion texture is an instantaneous motion map extracted from a dynamic texture. We observe that such motion maps exhibit values of two types: a discrete component at zero (absence of motion) and continuous motion values. We thus develop a mixed-state Markov random field model to represent motion textures. The core of our approach is to show that motion information is powerful enough to classify and segment dynamic textures if it is properly modeled regarding its specific nature and the local interactions involved. A parsimonious set of 11 parameters constitutes the descriptive feature of a motion texture. The motivation of the proposed formulation runs toward the analysis of dynamic video contents, and we tackle two related problems. First, we present a method for recognition and classification of motion textures, by means of the Kullback--Leibler distance between mixed-state statistical models. Second, we define a two-frame motion texture maximum a posteriori (MAP)-based segmentation method applicable to motion textures with deforming boundaries. We also investigate a new issue, the space-time dynamic texture segmentation, by combining the spatial segmentation and the recognition methods. Numerous experimental results are reported for those three problems which demonstrate the efficiency and accuracy of the proposed two-frame approach.

International audience

Dense point matching and tracking in image sequences is an open issue with implications in several domains, from content analysis to video editing. We observe that for long term dense point matching, some regions of the image are better matched by concatenation of consecutive motion vectors, while for others a direct long term matching is preferred. We propose a method to optimally estimate the correspondence of a point w.r.t. a reference image from a set of input motion estimations over different temporal intervals. Results on texture insertion by point tracking in the context of video editing are presented and compared with a state-of-the-art approach.

In this paper we address the problem of border detection for textured images by exploiting the concept of Bayesian complexity minimization as a generalization of the MDL paradigm in the Bayesian context. We aim at determining if a small window at a certain location corresponds to a single or several texture classes, which here are modeled as Gaussian Markov Random Fields (GMRF), thereby detecting the presence of borders. For doing this, a set of possible border configurations are tested by applying a Bayesian decision rule that includes two terms: a classical likelihood term related to the model fitting error, and a complexity penalizing term for the number and size of the window subdivisions in each configuration. The latter is derived from the Bayesian Information Criterion (BIC) for non-causal Markov models related to the MDL decision rule. Experiments on synthetic and real textured images segmentation support the approach with promising results.

Dynamic textures are time-varying visual patterns that exhibit certain spatio-temporal stationarity properties and are displayed mostly by natural scene elements. In this paper, we present new statistical models for the characterization of motion in this type of sequences. First we observe that motion measurements present values of two types: a discrete component at zero expressing the absence of motion and a continuous distribution for the rest of the motion values. Thus, we define random variables with mixed-states and propose to model a sequence of motion maps as a Markov chain, where the transition densities are mixed-state probability densities. Based on this approach, we propose a method for dynamic texture segmentation in real sequences showing the efficiency of the proposal in dynamic content analysis applications.

The aim of this work is to model the apparent motion in image sequences depicting natural dynamic scenes. We adopt the mixed-state Markov Random Fields (MRF) models recently introduced to represent so-called motion textures. The approach consists in describing the spatial distribution of some motion measurements which exhibit mixed-state nature: a discrete component related to the absence of motion and a continuous part for measurements different from zero. We propose several significative extensions to this model. We define an original motion texture segmentation method which does not assume conditional independence of the observations for each texture and normalizing factors are properly handled. Results on real examples demonstrate the accuracy and efficiency of our method.