Human beings have the ability to continuously analyze a video and immediately extract the motion components. We want to adopt this paradigm to provide a coherent and stable motion segmentation over the video sequence. In this perspective, we propose a novel long-term spatio-temporal model operating in a totally unsupervised way. It takes as input the volume of consecutive optical flow (OF) fields, and delivers a volume of segments of coherent motion over the video. More specifically, we have designed a transformer-based network, where we leverage a mathematically well-founded framework, the Evidence Lower Bound (ELBO), to derive the loss function. The loss function combines a flow reconstruction term involving spatio-temporal parametric motion models combining, in a novel way, polynomial (quadratic) motion models for the spatial dimensions and B-splines for the time dimension of the video sequence, and a regularization term enforcing temporal consistency on the segments. We report experiments on four VOS benchmarks, demonstrating competitive quantitative results while performing motion segmentation on a sequence in one go. We also highlight through visual results the key contributions on temporal consistency brought by our method.

Videomicroscopy is a promising tool combined with machine learning for studying the early development of in vitro fertilized bovine embryos and assessing its transferability as soon as possible. We aim to predict the embryo transferability within four days at most, taking 2D time-lapse microscopy videos as input. We formulate this problem as a supervised binary classification problem for the classes transferable and not transferable. The challenges are three-fold: 1) poorly discriminating appearance and motion, 2) class ambiguity, 3) small amount of annotated data. We propose a 3D convolutional neural network involving three pathways, which makes it multi-scale in time and able to handle appearance and motion in different ways. For training, we retain the focal loss. Our model, named SFR, compares favorably to other methods. Experiments demonstrate its effectiveness and accuracy for our challenging biological task.

Motion segmentation is one of the main tasks in computer vision and is relevant for many applications. The optical flow (OF) is the input generally used to segment every frame of a video sequence into regions of coherent motion. Temporal consistency is a key feature of motion segmentation, but it is often neglected. In this paper, we propose an original unsupervised spatiotemporal framework for motion segmentation from optical flow that fully investigates the temporal dimension of the problem. More specifically, we have defined a 3D network for multiple motion segmentation that takes as input a sub-volume of successive optical flows and delivers accordingly a sub-volume of coherent segmentation maps. Our network is trained in a fully unsupervised way, and the loss function combines a flow reconstruction term involving spatio-temporal parametric motion models, and a regularization term enforcing temporal consistency on the masks. We have specified an easy temporal linkage of the predicted segments. Besides, we have proposed a flexible and efficient way of coding U-nets. We report experiments on several VOS benchmarks with convincing quantitative results, while not using appearance and not training with any ground-truth data. We also highlight through visual results the distinctive contribution of the short- and long-term temporal consistency brought by our OF segmentation method.

In this paper, we present a CNN-based fully unsupervised method for motion segmentation from optical flow. We assume that the input optical flow can be represented as a piecewise set of parametric motion models, typically, affine or quadratic motion models. The core idea of our work is to leverage the Expectation-Maximization (EM) framework in order to design in a well-founded manner a loss function and a training procedure of our motion segmentation neural network that does not require either ground-truth or manual annotation. However, in contrast to the classical iterative EM, once the network is trained, we can provide a segmentation for any unseen optical flow field in a single inference step and without estimating any motion models. We investigate different loss functions including robust ones and propose a novel efficient data augmentation technique on the optical flow field, applicable to any network taking optical flow as input. In addition, our method is able by design to segment multiple motions. Our motion segmentation network was tested on four benchmarks, DAVIS2016, SegTrackV2, FBMS59, and MoCA, and performed very well, while being fast at test time.

In this paper, we are concerned with the detection of progressive dynamic saliency from video sequences. More precisely, we are interested in saliency related to motion and likely to appear progressively over time. It can be relevant to trigger alarms, to dedicate additional processing or to detect specific events. Trajectories represent the best way to support progressive dynamic saliency detection. Accordingly, we will talk about trajectory saliency. A trajectory will be qualified as salient if it deviates from normal trajectories that share a common motion pattern related to a given context. First, we need a compact while discriminative representation of trajectories. We adopt a (nearly) unsupervised learning-based approach. The latent code estimated by a recurrent auto-encoder provides the desired representation. In addition, we enforce consistency for normal (similar) trajectories through the auto-encoder loss function. The distance of the trajectory code to a prototype code accounting for normality is the means to detect salient trajectories. We validate our trajectory saliency detection method on synthetic and real trajectory datasets, and highlight the contributions of its different components. We compare our method favourably to existing methods on several saliency configurations constructed from the publicly available large dataset of pedestrian trajectories acquired in a railway station.

MOTIVATION: The revolution in light sheet microscopy enables the concurrent observation of thousands of dynamic processes, from single molecules to cellular organelles, with high spatiotemporal resolution. However, challenges in the interpretation of multidimensional data requires the fully automatic measurement of those motions to link local processes to cellular functions. This includes the design and the implementation of image processing pipelines able to deal with diverse motion types, and 3D visualization tools adapted to the human visual system. RESULTS: Here, we describe a new method for 3D motion estimation that addresses the aforementioned issues. We integrate 3D matching and variational approach to handle a diverse range of motion without any prior on the shape of moving objects. We compare different similarity measures to cope with intensity ambiguities and demonstrate the effectiveness of the Census signature for both stages. Additionally, we present two intuitive visualization approaches to adapt complex 3D measures into an interpretable 2D view, and a novel way to assess the quality of flow estimates in absence of ground truth. AVAILABILITY AND IMPLEMENTATION: https://team.inria.fr/serpico/data/3d-optical-flow-data/. SUPPLEMENTARY INFORMATION: Supplementary data are available at Bioinformatics online.

Rotoscoping, the detailed delineation of scene elements through a video shot, is a painstaking task of tremendous importance in professional post-production pipelines. While pixel-wise segmentation techniques can help for this task, professional rotoscoping tools rely on parametric curves that offer the artists a much better interactive control on the definition, editing and manipulation of the segments of interest. Sticking to this prevalent rotoscoping paradigm, we propose a novel framework to capture and track the visual aspect of an arbitrary object in a scene, given an initial closed outline of this object. This model combines a collection of local foreground/background appearance models spread along the outline, a global appearance model of the enclosed object and a set of distinctive foreground landmarks. The structure of this rich appearance model allows simple initialization, efficient iterative optimization with exact minimization at each step, and on-line adaptation in videos. We further extend this model by so-called trimaps which serve as an input to alpha-matting algorithms to allow truly seamless compositing. To this end, we leverage local classifiers attached to the roto-curves to define a confidence measure that is well-suited to define trimaps with adaptive band-widths. The resulting trimaps are parametric, temporally consistent and remain fully editable by the artist. We demonstrate qualitatively and quantitatively the merit of this framework through comparisons with tools based on either dynamic segmentation with a closed curve or pixel-wise binary labelling.

The paper addresses the problem of motion saliency in videos, that is, identifying regions that undergo motion departing from its context. We propose a new unsupervised paradigm to compute motion saliency maps. The key ingredient is the flow inpainting stage. Candidate regions are determined from the optical flow boundaries. The residual flow in these regions is given by the difference between the optical flow and the flow inpainted from the surrounding areas. It provides the cue for motion saliency. The method is flexible and general by relying on motion information only. Experimental results on the DAVIS 2016 benchmark demonstrate that the method compares favourably with state-of-the-art video saliency methods.

Detecting spot-like objects of different sizes in images is needed in many applications. Multiple image scales must then be handled for reliable spot segmentation. We define an original criterion based on the a contrario approach and the LoG scale-space framework to automatically select the meaningful scales. We then design a coarse-to-fine multi-scale spot segmentation scheme involving a locally adaptive thresholding across scales, to come up with the final map of segmented spots. We report experimental results on simulated and real images of different types, and we demonstrate that our method outperforms other existing methods.

BACKGROUND: Characterizing membrane dynamics is a key issue to understand cell exchanges with the extra-cellular medium. Total internal reflection fluorescence microscopy (TIRFM) is well suited to focus on the late steps of exocytosis at the plasma membrane. However, it is still a challenging task to quantify (lateral) diffusion and estimate local dynamics of proteins. RESULTS: A new model was introduced to represent the behavior of cargo transmembrane proteins during the vesicle fusion to the plasma membrane at the end of the exocytosis process. Two biophysical parameters, the diffusion coefficient and the release rate parameter, are automatically estimated from TIRFM image sequences, to account for both the lateral diffusion of molecules at the membrane and the continuous release of the proteins from the vesicle to the plasma membrane. Quantitative evaluation on 300 realistic computer-generated image sequences demonstrated the efficiency and accuracy of the method. The application of our method on 16 real TIRFM image sequences additionally revealed differences in the dynamic behavior of Transferrin Receptor (TfR) and Langerin proteins. CONCLUSION: An automated method has been designed to simultaneously estimate the diffusion coefficient and the release rate for each individual vesicle fusion event at the plasma membrane in TIRFM image sequences. It can be exploited for further deciphering cell membrane dynamics.

This paper considers the problem of localizing actions in videos as sequences of bounding boxes. The objective is to generate action proposals that are likely to include the action of interest, ideally achieving high recall with few proposals. Our contributions are threefold. First, inspired by selective search for object proposals, we introduce an approach to generate action proposals from spatiotemporal super-voxels in an unsupervised manner, we call them Tubelets . Second, along with the static features from individual frames our approach advantageously exploits motion. We introduce independent motion evidence as a feature to characterize how the action deviates from the background and explicitly incorporate such motion information in various stages of the proposal generation. Finally, we introduce spatiotemporal refinement of Tubelets, for more precise localization of actions, and pruning to keep the number of Tubelets limited. We demonstrate the suitability of our approach by extensive experiments for action proposal quality and action localization on three public datasets: UCF Sports, MSR-II and UCF101. For action proposal quality, our unsupervised proposals beat all other existing approaches on the three datasets. For action localization, we show top performance on both the trimmed videos of UCF Sports and UCF101 as well as the untrimmed videos of MSR-II.

The problem of localizing occlusions between consecutive frames of a video is important but rarely tackled on its own. In most works, it is tightly interleaved with the computation of accurate optical flows, which leads to a delicate chicken-and-egg problem. With this in mind, we propose a novel approach to occlusion detection where visibility or not of a point in next frame is formulated in terms of visual reconstruction. The key issue is now to determine how well a pixel in the first image can be "reconstructed" from co-located colors in the next image. We first exploit this reasoning at the pixel level with a new detection criterion. Contrary to the ubiquitous displaced-framedifference and forward-backward flow vector matching, the proposed alternative does not critically depend on a precomputed, dense displacement field, while being shown to be more effective. We then leverage this local modeling within an energy-minimization framework that delivers occlusion maps. An easy-to-obtain collection of parametric motion models is exploited within the energy to provide the required level of motion information. Our approach outperforms state-of-the-art detection methods on the challenging MPI Sintel dataset.

Parametric motion models are commonly used in image sequence analysis for different tasks. A robust estimation framework is usually required to reliably compute the motion model. The choice of the right model is also important. However, dealing simultaneously with both issues remains an open question. We propose a robust motion model selection method with two variants, which relies on the Fisher test. We also derive an interpretation of it as a robust Mallows' CP criterion. The resulting criterion is straightforward to compute. We have conducted a comparative experimental evaluation on different image sequences demonstrating the interest and the efficiency of the proposed method.

A number of applications in video analysis rely on a per-frame motion segmentation of the scene as key preprocessing step. Moreover, different settings in video production require extracting segmentation masks of multiple moving objects and object parts in a hierarchical fashion. In order to tackle this problem, we propose to analyze and exploit the compositional structure of scene motion to provide a segmentation which is not purely driven by local image information. Specifically, we leverage a hierarchical motion-based partition of the scene to capture a mid-level understanding of the dynamic video content. We present experimental results showing the strengths of this approach in comparison to current video segmentation approaches.

Accurately detecting subcellular particles in fluorescence microscopy is of primary interest for further quantitative analysis such as counting, tracking, or classification. Our primary goal is to segment vesicles likely to share nearly the same size in fluorescence microscopy images. Our method termed adaptive thresholding of Laplacian of Gaussian (LoG) images with autoselected scale (ATLAS) automatically selects the optimal scale corresponding to the most frequent spot size in the image. Four criteria are proposed and compared to determine the optimal scale in a scale-space framework. Then, the segmentation stage amounts to thresholding the LoG of the intensity image. In contrast to other methods, the threshold is locally adapted given a probability of false alarm (PFA) specified by the user for the whole set of images to be processed. The local threshold is automatically derived from the PFA value and local image statistics estimated in a window whose size is not a critical parameter. We also propose a new data set for benchmarking, consisting of six collections of one hundred images each, which exploits backgrounds extracted from real microscopy images. We have carried out an extensive comparative evaluation on several data sets with ground-truth, which demonstrates that ATLAS outperforms existing methods. ATLAS does not need any fine parameter tuning and requires very low computation time. Convincing results are also reported on real total internal reflection fluorescence microscopy images.

Image analysis applied to fluorescence live cell microscopy has become a key tool in molecular biology since it enables to characterize biological processes in space and time at the subcellular level. In fluorescence microscopy imaging, the moving tagged structures of interest, such as vesicles, appear as bright spots over a static or nonstatic background. In this paper, we consider the problem of vesicle segmentation and time-varying background estimation at the cellular scale. The main idea is to formulate the joint segmentation-estimation problem in the general conditional random field framework. Furthermore, segmentation of vesicles and background estimation are alternatively performed by energy minimization using a min cut-max flow algorithm. The proposed approach relies on a detection measure computed from intensity contrasts between neighboring blocks in fluorescence microscopy images. This approach permits analysis of either 2D + time or 3D + time data. We demonstrate the performance of the so-called C-CRAFT through an experimental comparison with the state-of-the-art methods in fluorescence video-microscopy. We also use this method to characterize the spatial and temporal distribution of Rab6 transport carriers at the cell periphery for two different specific adhesion geometries.

This paper considers the problem of action localization, where the objective is to determine when and where certain actions appear. We introduce a sampling strategy to produce 2D+t sequences of bounding boxes, called tubelets. Compared to state-of-the-art alternatives, this drastically reduces the number of hypotheses that are likely to include the action of interest. Our method is inspired by a recent technique introduced in the context of image localization. Beyond considering this technique for the first time for videos, we revisit this strategy for 2D+t sequences obtained from super-voxels. Our sampling strategy advantageously exploits a criterion that reflects how action related motion deviates from background motion. We demonstrate the interest of our approach by extensive experiments on two public datasets: UCF Sports and MSR-II. Our approach significantly outperforms the state-of-the-art on both datasets, while restricting the search of actions to a fraction of possible bounding box sequences.

Assessing crowd behaviors from videos is a difficult task while of interest in many applications. We have defined a novel approach which identifies from two successive frames only, crowd behaviors expressed by simple image motion patterns. It relies on the estimation of a collection of sub-affine motion models in the image, a local motion classification based on a penalized likelihood criterion, and a regularization stage involving inhibition and reinforcement factors. We have also developed an original and simple method for recovering the dominant paths followed by people in the observed scene. It involves the introduction of local paths determined from the space-time average of the parametric motion subfields selected in each image block. Experiments on synthetic and real scenes have demonstrated the performance of our method.

Recognizing dynamic behaviors of dense crowds in videos is of great interest in many surveillance applications. In contrast to most existing methods which are based on trajectories or tracklets, our approach for crowd motion analysis provides a crowd motion classification on a frame-by-frame and pixel-wise basis. Indeed, we only compute affine motion models from pairs of two consecutive video images. The classification itself relies on simple rules on the coefficients of the computed affine motion models, and therefore does not imply any prior learning stage. The overall method proceeds in four steps: (i) detection of moving points, (ii) computation of a set of motion model candidates over a collection of windows, (iii) selection of the best motion model at each point owing to a maximum likelihood criterion, (iv) determination of the crowd motion class at each pixel with a hierarchical classification tree regularized by majority votes. The algorithm is almost parameter-free, and is efficient in terms of memory and computation load. Experiments on computer-generated sequences and real video sequences demonstrate that our method is accurate, and can successfully handle complex situations.

Several recent works on action recognition have attested the importance of explicitly integrating motion characteristics in the video description. This paper establishes that adequately decomposing visual motion into dominant and residual motions, both in the extraction of the space-time trajectories and for the computation of descriptors, significantly improves action recognition algorithms. Then, we design a new motion descriptor, the DCS descriptor, based on differential motion scalar quantities, divergence, curl and shear features. It captures additional information on the local motion patterns enhancing results. Finally, applying the recent VLAD coding technique proposed in image retrieval provides a substantial improvement for action recognition. Our three contributions are complementary and lead to outperform all reported results by a significant margin on three challenging datasets, namely Hollywood 2, HMDB51 and Olympic Sports.

A motion texture is an instantaneous motion map extracted from a dynamic texture. We observe that such motion maps exhibit values of two types: a discrete component at zero (absence of motion) and continuous motion values. We thus develop a mixed-state Markov random field model to represent motion textures. The core of our approach is to show that motion information is powerful enough to classify and segment dynamic textures if it is properly modeled regarding its specific nature and the local interactions involved. A parsimonious set of 11 parameters constitutes the descriptive feature of a motion texture. The motivation of the proposed formulation runs toward the analysis of dynamic video contents, and we tackle two related problems. First, we present a method for recognition and classification of motion textures, by means of the Kullback--Leibler distance between mixed-state statistical models. Second, we define a two-frame motion texture maximum a posteriori (MAP)-based segmentation method applicable to motion textures with deforming boundaries. We also investigate a new issue, the space-time dynamic texture segmentation, by combining the spatial segmentation and the recognition methods. Numerous experimental results are reported for those three problems which demonstrate the efficiency and accuracy of the proposed two-frame approach.

We present a nonparametric regression method for denoising 3-D image sequences acquired via fluorescence microscopy. The proposed method exploits the redundancy of the 3-D+time information to improve the signal-to-noise ratio of images corrupted by Poisson-Gaussian noise. A variance stabilization transform is first applied to the image-data to remove the dependence between the mean and variance of intensity values. This preprocessing requires the knowledge of parameters related to the acquisition system, also estimated in our approach. In a second step, we propose an original statistical patch-based framework for noise reduction and preservation of space-time discontinuities. In our study, discontinuities are related to small moving spots with high velocity observed in fluorescence video-microscopy. The idea is to minimize an objective nonlocal energy functional involving spatio-temporal image patches. The minimizer has a simple form and is defined as the weighted average of input data taken in spatially-varying neighborhoods. The size of each neighborhood is optimized to improve the performance of the pointwise estimator. The performance of the algorithm (which requires no motion estimation) is then evaluated on both synthetic and real image sequences using qualitative and quantitative criteria.

The present work concerns the analysis of dynamic scenes from earth observation images. We are interested in building a map which, on one hand locates places of change, on the other hand, reconstructs a unique visual information of the non-change areas. We show in this paper that such a problem can naturally be takled with conditional mixed-state random field modeling (mixed-state CRF), where the ¿mixed state¿ refers to the symbolic or continous nature of the unknown variable. The maximum a posteriori (MAP) estimation of the CRF is, through the Hammersley-Clifford theorem, turned into an energy minimisation problem. We tested the model on several Quickbird images and illustrate the quality of the results.

Dynamic textures are time-varying visual patterns that exhibit certain spatio-temporal stationarity properties and are displayed mostly by natural scene elements. In this paper, we present new statistical models for the characterization of motion in this type of sequences. First we observe that motion measurements present values of two types: a discrete component at zero expressing the absence of motion and a continuous distribution for the rest of the motion values. Thus, we define random variables with mixed-states and propose to model a sequence of motion maps as a Markov chain, where the transition densities are mixed-state probability densities. Based on this approach, we propose a method for dynamic texture segmentation in real sequences showing the efficiency of the proposal in dynamic content analysis applications.

This paper presents a content-based approach for temporal segmentation of videos. Tracked objects are characterized by their 2D trajectories which are used in a meaningful way to model visual semantics, i.e., the observed single video object activities and their interactions. To this end, hierarchical Semi-Markov Chains (SMCs) are computed in order to take into account the temporal causalities of object motions. Object movements are characterized using local invariant features computed from the curvature and velocity values while interactions are represented by the temporal evolution of the distance between objects. We have evaluated our method on squash video sequences, and have favorably compared with other methods including Hidden Markov Models (HMMs).

This work is dedicated to a statistical trajectory-based approach addressing two issues related to dynamic video content understanding: recognition of events and detection of unexpected events. Appropriate local differential features combining curvature and motion magnitude are defined and robustly computed on the motion trajectories in the image sequence. These features are invariant to image translation, in-the-plane rotation and spatial scaling. The temporal causality of the features is then captured by hidden Markov models dedicated to trajectory description, whose states are properly quantized values. The similarity between trajectories is expressed by exploiting this quantization-based HMM framework. Moreover statistical techniques have been developed for parameter estimations. Evaluations of the method have been conducted on several data sets including real trajectories obtained from sport videos, especially Formula One and ski TV program. The novel method compares favorably with other methods including feature histogram comparisons, HMM/GMM modeling and SVM classification.

This paper describes an original method for classifying object motion trajectories in video sequences in order to recognize dynamic events. Similarities between trajectories are expressed from hidden Markov models representing each trajectory. We have favorably compared our method to several other ones, including histogram comparison, longest common subsequence distance and SVM classification. Trajectory features are computed from the curvature and velocity values at each point of the trajectory, so that they are invariant to translation, rotation and scale. We have evaluated our method on two sets of data, a first one composed of typical classes of synthetic trajectories (such as parabola or clothoid), and a second one formed with trajectories obtained by tracking cars in a Formula 1 race video.

This paper presents a method for detecting independent temporally-persistent motion patterns in image sequences. The result is a description of the dynamic content of a video sequence in terms of moving objects, their number, image position and approximate motion. For each detected motion pattern a local trajectory as well as a confidence level is provided. The method is based on local motion measurements extracted from short video segments. These measurements are mapped in an adequate grouping space where independent trajectories correspond to distinct clusters. The automatic cluster detection is handled in an a contrario framework, which is general and involves no parameter tuning. The method was validated on real video sequences featuring rigid and non-rigid moving objects, static and mobile cameras, and distracting motions. The output of this method could initialize tracking algorithms. Applications of interest are robot navigation, car-driver assistance, surveillance and activity recognition.

We present a novel space-time patch-based method for image sequence restoration. We propose an adaptive statistical estimation framework based on the local analysis of the bias-variance trade-off. At each pixel, the space-time neighborhood is adapted to improve the performance of the proposed patch-based estimator. The proposed method is unsupervised and requires no motion estimation. Nevertheless, it can also be combined with motion estimation to cope with very large displacements due to camera motion. Experiments show that this method is able to drastically improve the quality of highly corrupted image sequences. Quantitative evaluations on standard artificially noise-corrupted image sequences demonstrate that our method outperforms other recent competitive methods. We also report convincing results on real noisy image sequences.

This paper is concerned with motion estimation in transparent X-ray image sequences. Most of these medical images can be divided into areas containing at most two moving transparent layers. We will call it bi-distributed transparency. The first contribution of this paper is a motion estimation framework for the two-layer transparency case, able to handle noisy and low-contrasted X-ray image sequences. It involves three steps: block-matching, affine fit and gradient-based parametric estimation. This estimation scheme is then extended to the bi-distributed transparency case. The second step is now formulated as a joint motion segmentation-estimation problem solved by the iterative minimization of a MRF-based energy function. This framework has been applied to synthetic and real image sequences with quite satisfactory results.

This paper describes a method for separating moving objects from temporally varying background in time-lapse confocal microscopy image sequences representing fluorescently tagged moving vesicles. A temporal linear model is considered for background modeling whose parameters are robustly estimated using asymmetric M-estimators combined with a bias-variance trade-off criterion. Furthermore, we propose an original approach for automatically detecting moving objects in the image sequence. Experimental results demonstrate the interest of this proposed method which can be relevant for biological studies from image sequences.

The aim of this work is to model the apparent motion in image sequences depicting natural dynamic scenes. We adopt the mixed-state Markov Random Fields (MRF) models recently introduced to represent so-called motion textures. The approach consists in describing the spatial distribution of some motion measurements which exhibit mixed-state nature: a discrete component related to the absence of motion and a continuous part for measurements different from zero. We propose several significative extensions to this model. We define an original motion texture segmentation method which does not assume conditional independence of the observations for each texture and normalizing factors are properly handled. Results on real examples demonstrate the accuracy and efficiency of our method.

The exploitation of video data requires methods able to extract high-level information from the images. Video summarization, video retrieval, or video surveillance are examples of applications. In this paper, we tackle the challenging problem of recognizing dynamic video contents from low-level motion features. We adopt a statistical approach involving modeling, (supervised) learning, and classification issues. Because of the diversity of video content (even for a given class of events), we have to design appropriate models of visual motion and learn them from videos. We have defined original parsimonious global probabilistic motion models, both for the dominant image motion (assumed to be due to the camera motion) and the residual image motion (related to scene motion). Motion measurements include affine motion models to capture the camera motion and low-level local motion features to account for scene motion. Motion learning and recognition are solved using maximum likelihood criteria. To validate the interest of the proposed motion modeling and recognition framework, we report dynamic content recognition results on sports videos.

Because of their lack of rules, general broadcast videos are more difficult to analyze than news or sport videos. To retrieve human interventions in this context, a robust face tracker is needed. The approach we investigate for face tracking combines three main modules that are a face detector, a region-based tracker and an eye tracker. The region-based tracker relies on a robust parametric motion estimation technique. The eye tracker is based on a Kalman filter. The analysis of the coherence of the trackers output provides an efficient way to detect profile positions and tracking errors. We have thus defined an entirely automatic tracker, able to manage several appearing/disappearing faces, without any a priori knowledge on the image sequence. Experimental results on broadcast videos demonstrate its efficiency to deal with large and rapid motions, occlusions and faces in profile position.

A new framework dealing with motion estimation in transparent images is presented. It relies on a block-oriented estimation involving an efficient multiresolution minimization. A downhill simplex method provides an appropriate initialization to this scheme. The estimated velocity vectors are greatly improved by an original postprocessing stage which performs a single motion estimation on differences of warped images. Finally, a regularization step is carried out. It is demonstrated on a large set of simulations that a quarter-pixel accuracy can be attained on noise-free images. The case of noisy images is also addressed and provides satisfactory results, even in the case of low-contrasted medical images. An example on real clinical images is also reported with promising results.

Visual tracking is still a challenging problem in computer vision. The applications of visual tracking are far-reaching, ranging from surveillance and monitoring to smart rooms. In this work, we propose a new method to track arbitrary objects using both sum-of-squared differences (SSD) and color-based mean-shift (MS) trackers in the Kalman filter framework. The SSD and the MS trackers complement each other by overcoming their respective disadvantages. The rapid model change in SSD tracker is overcome by the MS tracker module, while the inability of MS tracker to handle large displacements and occlusions is circumvented by the SSD module. In addition, rapid scale changes of the object generated by camera ego-motion or zooming are measured by a global affine motion estimation. Finally, the global appearance model on which MS relies is updated, based on the Bhattacharyya distance between this target model and current candidate model. This permits to tackle global appearance changes of the object. The performance of the proposed tracker is better than the individual SSD and MS trackers.

We present a spatio-temporal filtering method for significantly increasing the signal-to-noise ratio in noisy fluorescence microscopic image sequences where small particles have to be tracked from frame to frame. Image sequences restoration is achieved using a spatio-temporal adaptive window approach with an appropriate on-line window geometry specification. We have applied this method to noisy synthetic and real 3D image sequences where a large number of small fluorescently labelled vesicles are moving in regions close to the Golgi apparatus. The SNR is shown to be drastically improved and the enhanced vesicles can be segmented. This novel approach can be further used for biological studies where the dynamic of small objects of interest has to be analyzed in molecular and sub-cellular bio-imaging.

A contrario modeling is a detection framework based on a perceptual grouping principle. Based on an existing a contrario motion detection method, this paper presents two solutions for improving detection results by using the confidence levels issued from the a contrario testing framework. The first one embeds the confidence levels in a maximality principle in order to operate a sensible selection among detections. This improves the description of the moving objects. The second extension enables to take into account information coming from more than one residual motion map in order to improve detection results on small slowly moving objects.

We present a soccer video abstraction method based on the analysis of the audio and video streams. This method could be applied to other sports as rugby or american football. The main contribution of this paper is the design of an unsupervised summarization method, and more specifically, the introduction of an efficient detector of excited speech segments. An excited commentary is supposed to correspond to an interesting moment of the game. It is simultaneously characterized by an increase of the pitch (or fundamental frequency) within the voiced segments and an increase of the energy supported by the harmonics of the pitch. The pitch is estimated from the autocorrelation function and its local increases are detected from a multiresolution technique. We introduce a specific energy measure for the voiced segments. A statistical analysis of the energy measures is performed to detect the most excited parts of the speech. A deterministic combination of excited speech detection, dominant color identification and camera motion analysis is then performed in order to discriminate between excited speech sequences of the game and excited speech sequences in commercials or in studio shots included in the processed TV programs.The method presented here does not need any learning stage. It has been tested on seven soccer videos for a total duration of almost 20 hours.

We propose a context-based model of video abstraction exploiting both audio and video features and applied to tennis TV programs. We can automatically produce different types of summary of a given video depending on the users' constraints or preferences. We have first designed an efficient and accurate temporal segmentation of the video into segments homogeneous w.r.t the camera motion. We introduce original visual descriptors related to the dominant and residual image motions. The different summary types are obtained by specifying adapted classification criteria which involve audio features to select the relevant segments to be included in the video abstract. The proposed scheme has been validated on 22 hours of tennis videos.

We present a supervised method for the detection and retrieval of relevant events in videos according to dynamic content. We adopt a statistical representation where residual and camera motion informations are characterized by probabilistic models. In an off-line stage, the models associated to pre-identified classes of meaningful dynamic events are learned from a given training set of video samples. Then, a classification and selection algorithm is applied on each segment of a temporal segmentation of the video to process, by exploiting this statistical framework. Only the segments associated to classes defined as relevant in terms of dynamic event can then be selected. The efficiency of the proposed method is evaluated on sport videos for which categories of relevant events can be explicitly defined.

We present a method for motion-based video segmentation and segment classification as a step towards video summarisation. The sequential segmentation of the video is performed by detecting changes in the dominant image motion, assumed to be related to camera motion. It is achieved by analysing the temporal variations of coefficients of the global 2D affine motion model (robustly) estimated. The obtained video segments supply reasonable temporal regions to apply a classification algorithm. To this end, we adopt a statistical representation of the residual motion content of the video scene, relying on the distribution of temporal cooccurrences of local motion-related measurements. Pre-identified classes of dynamic events are learned off-line from a training set of video samples of the genre of interest. Each video segment is then classified according to a maximum likelihood (ML) principle. Finally, excerpts of the relevant classes can be selected for video summarisation.

A new approach for motion characterization in image sequences is presented. It relies on the probabilistic modeling of temporal and scale co-occurrence distributions of local motion-related measurements directly computed over image sequences. Temporal multiscale Gibbs models allow us to handle both spatial and temporal aspects of image motion content within a unified statistical framework. Since this modeling mainly involves the scalar product between co-occurrence values and Gibbs potentials, we can formulate and address several fundamental issues: model estimation according to the ML criterion (hence, model training and learning) and motion classification. We have conducted motion recognition experiments over a large set of real image sequences comprising various motion types such as temporal texture samples, human motion examples, and rigid motion situations.

We present in this paper an original approach for content-based video segmentation using motion information. The method is generic and does not require any knowledge about the type of the processed video. Its relies on the analysis of the temporal evolution of the dynamic content of the video.

The early and accurate segmentation of low clouds during the night-time is an important task for nowcasting. It requires that observations can be acquired at a sufficient time rate as provided by the geostationary METEOSAT satellite over Europe. However, the information supplied by the single infrared METEOSAT channel available by night is not sufficient to discriminate between low clouds and ground during night from a single image. To tackle this issue, the authors consider several sources of information extracted from an infrared image sequence. Indeed, they exploit both relevant local motion-based measurements, intensity images and thermal parameters estimated over blocks, along with local contextual information. A statistical contextual labeling process in two classes, involving "low clouds" and "clear sky," is performed on the warmer pixels. It is formulated within a Bayesian estimation framework associated with Markov random field (MRF) models. This comes to minimize a global energy function comprising three terms: two data-driven terms (thermal and motion-based ones) and a regularization term expressing a priori knowledge on the label field (expected spatial contextual properties). The authors propose a progressive minimization procedure of this energy function starting from initial reliably labeled pixels and involving only local computation.

This paper describes an original approach for content-based video indexing and retrieval. We aim at providing a global interpretation of the dynamic content of video shots without any prior motion segmentation and without any use of dense optic flow fields. To this end, we exploit the spatio-temporal distribution, within a shot, of appropriate local motion-related measurements derived from the spatio-temporal derivatives of the intensity function. These distributions are then represented by causal Gibbs models. To be independent of camera movement, the motion-related measurements are computed in the image sequence generated by compensating the estimated dominant image motion in the original sequence. The statistical modeling framework considered makes the exact computation of the conditional likelihood of a video shot belonging to a given motion or more generally to an activity class feasible. This property allows us to develop a general statistical framework for video indexing and retrieval with query-by-example. We build a hierarchical structure of the processed video database according to motion content similarity. This results in a binary tree where each node is associated to an estimated causal Gibbs model. We consider a similarity measure inspired from Kullback-Leibler divergence. Then, retrieval with query-by-example is performed through this binary tree using the maximum a posteriori (MAP) criterion. We have obtained promising results on a set of various real image sequences.

We present an original approach for non parametric motion analysis in image sequences. It relies on the statistical modeling of distributions of local motion-related measurements computed over image sequences. Contrary to previously proposed methods, the use of temporal multiscale Gibbs models allows us to handle in a unified statistical framework both spatial and temporal aspects of motion content. The important feature of our probabilistic scheme is to make the exact computation of conditional likelihood functions feasible and simple. It enables us to straightforwardly achieve model estimation according to the ML criterion and to benefit from a statistical point of view for classification issues. We have conducted motion recognition experiments over a large set of real image sequences comprising various motion types such as temporal texture samples, human motion examples and rigid motion situations.

This paper deals with the detection of moving objects in videos, directly from MPEG coded data, with a view to content-based video indexing. The detection is stated as a Markovian labeling issue in terms of macroblocks conforming or not to the estimated dominant image motion assumed to be due to the camera motion. The dominant motion estimation and the moving object detection stages only utilize MPEG motion vectors and DC coefficients of the discrete cosine transform (DCT) directly extracted from the MPEG bit stream of the processed video. Therefore, our method implies a very low computational cost. Experimental results have demonstrated the interest of the proposed approach.

This paper describes an original approach for non parametric motion analysis in image sequences. It relies on a statistical modeling of distributions of local motion-related measurements, computed over image sequences, resulting from spatio-temporal random walks. It handles in a single probabilistic framework both spatial and temporal properties of motion content. The important feature of our method is to make feasible the exact computation of conditional likelihood functions. We have carried out motion recognition experiments over a large set of real image sequences comprising various motion types.

This paper presents the determination of bathymetric maps from the analysis of multispectral SPOT images. To this end, we have developed a multispectral segmentation method based on a hierarchical Markovian modeling including the unsupervised estimation of the model parameters. In each segmented region, an adaptive bathymetric inversion model is then applied in order to recover the water depth (mainly in coastal areas). Bathymetric estimation has been validated on real data, for which control points are available and correspond to bathymetric measures supplied by previous hydrographic campaigns.

This paper deals with the use of a 2D parametric motion model in a spatio-temporal filtering scheme to reduce noise in image sequences. We estimate with a robust method an affine motion model accounting for the dominant image motion. Then, we cancel it before applying an adaptive spatiotemporal filter. We have compared the performance of several filtering techniques and evaluated the influence of the motion compensation step on this performance.

We propose a low cost scheme for reconstructing high resolution images from noisy, and eventually blurred image sequences. The super-resolution is achieved by an iterative back projection method. To account for noise in image sequence, we first apply a spatio-temporal Wiener filter computed via a 3D DFT. In the filtering process, we need to compensate for apparent motion to ensure proper results. Furthermore, the knowledge of subpixel motion is necessary for super-resolution. In both cases, we exploit a parametric motion model to keep a good trade-off between accuracy and computation time.

In this paper we propose an original approach for content-based video indexing and retrieval. It relies on the tracking of entities of interest and the analysis of their apparent motion. To characterize the dynamic information attached to these objects, we consider a probabilistic modeling of the spatio-temporal distribution of the optic flow field computed within the tracked area after canceling the estimated dominant motion due to camera movement. This leads to a general statistical framework for motion-based video classification and retrieval. We have obtained promising results on a set of various real image sequences.

Segmenting video documents into sequences from elementary shots to supply an appropriate higher level description of the video is a challenging task. The paper presents a two-stage method. First, we build a binary agglomerative hierarchical time-constrained shot clustering. Second, based on the cophenetic criterion, a breaking distance between shots is computed to detect sequence changes. Various options are implemented and compared. Real experiments have proved that the proposed criterion can be efficiently used to achieve appropriate segmentation into sequences.

We present an original statistical classification method using a deformable template model to separate natural objects from man-made objects in an image provided by a high resolution sonar. A prior knowledge of the manufactured object shadow shape is captured by a prototype template, along with a set of admissible linear transformations, to take into account the shape variability. Then, the classification problem is defined as a two-step process: 1) the detection problem of a region of interest in the input image is stated as the minimization of a cost function; and 2) the value of this function at convergence allows one to determine whether the desired object is present or not in the sonar image. The energy minimization problem is tackled using relaxation techniques. In this context, we compare the results obtained with a deterministic relaxation technique and two stochastic relaxation methods: simulated annealing and a hybrid genetic algorithm. This latter method has been successfully tested on real and synthetic sonar images, yielding very promising results.

This paper addresses the problem of analyzing human motion in an image sequence. This is of particular importance for video-surveillance applications. We have developed an original and efficient approach to track the apparent contours of a moving articulated structure, avoiding the use of 3D models. This method exploits spatio-temporal XT-slices from the image sequence volume XYT, where typical trajectory patterns can be associated with articulated motion such as human walking. We reconstruct these trajectories online by introducing an appropriate predictive model while correctly handling occlusion periods. This paradigm can lead to a simple trajectory recognition scheme. Experiments with real-world images depicting human walk or gesture are reported, and obtained results validate the proposed approach.

This paper is concerned with hierarchical Markov random field (MRP) models and their application to sonar image segmentation. We present an original hierarchical segmentation procedure devoted to images given by a high-resolution sonar. The sonar image is segmented into two kinds of regions: shadow (corresponding to a lack of acoustic reverberation behind each object lying on the sea-bed) and sea-bottom reverberation. The proposed unsupervised scheme takes into account the variety of the laws in the distribution mixture of a sonar image, and it estimates both the parameters of noise distributions and the parameters of the Markovian prior. For the estimation step, we use an iterative technique which combines a maximum likelihood approach (for noise model parameters) with a least-squares method (for MRF-based prior). In order to model more precisely the local and global characteristics of image content at different scales, we introduce a hierarchical model involving a pyramidal label field. It combines coarse-to-fine causal interactions with a spatial neighborhood structure. This new method of segmentation, called the scale causal multigrid (SCM) algorithm, has been successfully applied to real sonar images and seems to be well suited to the segmentation of very noisy images. The experiments reported in this paper demonstrate that the discussed method performs better than other hierarchical schemes for sonar image segmentation.

This paper presents the dynamic scene analysis part of an original and consistent framework to video partitioning into shots, camera motion estimation and multiple motion analysis with a view to content-based video indexing. All the information parts required to achieve these different goals result from handling the apparent motion within consecutive image pairs. Within each extracted shot, a binary segmentation of the image is performed into regions whose motion either conforms or not to the 2D estimated dominant motion represented by a quadratic motion model. This paper focuses on a low-cost method based on projective geometry criteria to distinguish non-conforming regions generated by really moving objects from static ones in the scene. The proposed algorithm is validated on a variety of real image sequences.

We present an original method for tracking, in an image sequence, complex objects which can be approximately modeled by a polyhedral shape. The approach relies on the estimation of the 2D object image motion along with the computation of the 3D object pose. The proposed method fulfills real-time constraints along with reliability and robustness requirements. Real tracking experiments and results concerning a visual servoing positioning task are presented.

We aim at detecting moving objects in color image sequences acquired with a mobile camera. This issue is of key importance in many application fields. To accurately recover motion boundaries, we exploit a fine spatial image partition supplied by a MRF-based color segmentation algorithm. We introduce a region-level graph modeling embedded in a Markovian framework to detect moving objects in the scene viewed by a mobile camera. This is stated as the binary segmentation into regions conforming or not conforming to the dominant image motion assumed to be due to the camera movement. The method is validated on real image sequences.

We present an original method for tracking, in an image sequence, complex objects which can be modeled approximately by a polyhedral shape. The approach relies on the estimation of the object image motion as well as the computation of the object pose. The proposed method fulfills real-time constraints along with reliability and robustness requirements.

This paper presents an unsupervised image segmentation method with applications to ocean cartography. By using SPOT satellite data, the aim is to improve the automatic production of bathymetric charts. Indeed, in coastal areas, satellite images provide the radiometry of the electromagnetic waves backscattered by the vegetation, the sea or the sea floor depending on the sea depth in littoral areas. The proposed segmentation method is based on a hierarchical Markovian model defined on a quad-tree, combining multispectral data. One of its interests is to take explicitly into account the correlation between the three spectral channels of the observation. Classification results obtained with synthetic and real images demonstrate the efficiency of the method.

This paper describes an original approach to partitioning of a video document into shots. Instead of an interframe similarity measure which is directly intensity based, we exploit image motion information, which is generally more intrinsic to the video structure itself. The proposed scheme aims at detecting all types of transitions between shots using a single technique and the same parameter set, rather than a set of dedicated methods. The proposed shot change detection method is related to the computation, at each time instant, of the dominant image motion represented by a two-dimensional affine model. More precisely, we analyze the temporal evolution of the size of the support associated to the estimated dominant motion. Besides, the computation of the global motion model supplies by-products, such as qualitative camera motion description, which we describe in this paper, and other possible extensions, such as mosaicing and mobile zone detection. Results on videos of various content types are reported and validate the proposed approach.

This paper is concerned with the analysis of 2D fluid motion from numerical images. The interpretation of such deformable flow fields can be derived from the characterization of linear motion models provided that first order approximations are considered in an adequate neighborhood of so-called singular points where the velocity becomes null. However, locating such points, delimiting this neighborhood, and estimating the associated 2D affine motion model, are intricate difficult problems. We explicitly address these three joint problems according to a statistical adaptive approach. In the fluid mechanics images we are dealing with, the motion model can be directly inferred from a single image, since the visualized form accounts for the underlying motion. We have developed an original method which relies on an orthogonality constraint between the spatial image gradient field and the motion model velocity field, while explicitly formalizing and handling both model and measurement noises. This method has been validated on several real fluid flow images.

We present a new classification method using a deformable template model to separate natural objects from man made objects in an image given by a high resolution sonar. A prior knowledge of the manufactured object shadow shape is described by a prototype template and a set of admissible linear transformations to take into account the shape variability. Then, the classification problem is defined as a two step process; firstly the detection problem of a region of interest in the input image is stated in a Bayesian framework and is posed as an equivalent energy minimization problem of an objective function: in this paper, this energy minimization problem is solved by using a hybrid genetic algorithm (GA). Secondly, the value of this function at convergence allows one to determine the presence of the desired object in the sonar image. This method has been successfully tested on real and synthetic sonar images, yielding very promising results.

This paper describes an original approach for motion interpretation with a view to content-based video indexing. We exploit a statistical analysis of the temporal distribution of appropriate local motion-based measures to perform a global motion characterization. We consider motion features extracted from temporal cooccurrence matrices, and related to properties of homogeneity, acceleration or complexity. Results on various real video sequences are reported and provide a first validation of the approach.

Efficient real-time robotic tasks using a monocular vision system were previously developed with simple objects (e.g. white points on a black background), within a visual servoing context. Due to recent developments, it is now possible to design real-time visual tasks exploiting motion information in the image, estimated by robust algorithms. This paper proposes such an approach to track complex objects, such as a pedestrian. It consists in integrating the measured 2D motion of the object to recover its 2D-position in the image. The principle of the tracking task is to control the camera pan and tilt such that the estimated center of the object appears at the center of the image. Real-time experimental results demonstrate the efficiency and the robustness of the method.

We propose a 2D visual motion estimation method which can be exploited to achieve a dynamic positioning (e.g. by gaze control) with respect to a sea-bottom area of interest of a video camera mounted on a subsea vehicle. It mainly involves a dominant 2D motion robust estimation step from underwater video sequences. Optimizations carried out on the motion estimation code have made possible the use of our algorithm in "application-related real-time" for scientific exploration or inspection tasks. We have developed a friendly and efficient interface to perform this algorithm in an operational context. Experiments dealing with complex real underwater scenes are reported and validate the approach.

This paper presents the first version of our platform for automatically building the structure of a video sequence. The first application uses semi-automatic tools based only on image analysis for building interactive videos: decomposing the video into shots, extracting and tracking objects within each shot and linking occurrences of similar objects among the shots. The second application provides the end user with a powerful browser to navigate through any preprocessed hypervideo.

This paper deals with the problem of motion-based segmentation of image sequences. Such partitions are multiple-purpose in dynamic scene analysis. We first extract a spatial texture-based partition using an unsupervised MRF approach. The regions obtained are then grouped according to a motion-based criterion. This grouping process relies on two motion estimation techniques and exploits centextual information between regions. In contrast with clustering techniques, region grouping is formalized as a motion-based graph labeling process, within a Markovian framework. Results on real-world image sequences are shown and validate the proposed method.

This work deals with unsupervised sonar image segmentation. We present a new estimation segmentation procedure using the an iterative method called iterative conditional estimation (ICE). This method takes into account the variety of the laws in the distribution mixture of a sonar image and the estimation of the parameters of the label field (modeled by a Markov random field (MRF)). For the estimation step we use a maximum likelihood estimation for the noise model parameters and the least square method proposed by Derin et al. (1987) to estimate the MRF prior model. Then, in order to obtain a good segmentation and to speed up the convergence rate, we use a multigrid strategy with the previously estimated parameters. This technique has been successfully applied to real sonar images and is compatible with an automatic treatment of massive amounts of data.

This paper is concerned with the problem of analysing articulated motion in image sequences, especially applied to case of the human motion. We have developed an approach to track the contours of a moving articulated structure, designed in a purely 2D+t form avoiding explicit and dedicated models. This method exploits spatio-temporal XT-slices of the image sequence volume XYT, where articulated motion, such as the gait of a walker, turns out to reveal typical trajectory providing relevant signatures. We propose a reconstruction algorithm of these significant trajectories, operating on-line and resorting to an original predictive model in order to deal with occultation periods. This paradigm provides the basis of trajectory classification and recognition schemes. Results are presented for real-world images depicting human movement of gestures and walking.

This paper describes an original approach which jointly addresses the two issues of video partitioning and camera motion characterization in the context of content-based video indexing. It can cope with scenes containing moving objects. Detection of shot changes and recognition of the movements of the system-of-view are both derived from the computation, at each time instant, of the dominant motion in the image represented by a 2D affine model, and from the variation of the size of its associated support. The successive steps of the method rely on statistical techniques ensuring robustness and efficiency. Results on a real documentary video are reported and validate the proposed approach.

This paper deals with sonar image segmentation based on a hierarchical Markovian modeling. The designed Markov random field (MRF) model takes into account both the phenomenon of speckle noise through Rayleigh's law, and notions of geometry related to the shape of object shadows. We adopt an 8-connexity neighbourhood in order to discriminate geometric and non-regular shadows. MRF are well adapted for this kind of segmentation where a priori knowledge about the shapes we are searching is available. Besides, the introduced hierarchical modeling allows us to successfully improve the sonar image segmentation while speeding up the iterative optimization scheme.

In this paper we address an important issue in motion analysis: the detection of moving objects. A statistical approach is adopted in order to formulate the problem. The inter-frame difference is modeled by a mixture of Laplacian distributions, and a Gibbs random field is used for describing the label set. A new method to determine the regularization parameter is proposed, based on a voting technique. Then two different multiscale algorithms are evaluated, and the labeling problem is solved using either ICM (iterated conditional modes) or HCF (highest confidence first) algorithms. Experimental results are provided using synthetic and real video sequences.

This paper deals with the determination of the main components of an outdoor scene from an image sequence observed by a mobile camera. By components, we mean the different depth "layers" of the scene. To segment the scene, we exploit the 2D motion which implicitly contains relative depth information. To achieve this segmentation, 2D affine motion models are considered. Models parameters for each extracted region are estimated from a dense velocity field. Its computation relies on a nonlinear diffusion method which preserves the motion discontinuities and supplies a consistency measure map. These data are used as observations in a hierarchical approach composed of two levels. The local merging step which classifies the pixels into regions, and the global merging step which ensures the consistency of each extracted region. The local merging step is embedded in a Markov random fields formalism, whereas the global merging step is also based on an energy formulation.

This paper deals with the recovery of 3D information using a single mobile camera in the context of active vision. First, we propose a general revisited formulation of the structure-from-known-motion issue. Within the same formalism, we handle various kinds of 3D geometrical primitives such as points, lines, cylinders, spheres, etc. We also aim at minimizing effects of the different measurement errors which are involved in such a process. More precisely, we mathematically determine optimal camera configurations and motions which lead to a robust and accurate estimation of the 3D structure parameters. We apply the visual servoing approach to perform these camera motions using a control law in closed-loop with respect to visual data. Real-time experiments dealing with 3D structure estimation of points and cylinders are reported. They demonstrate that this active vision strategy can very significantly improve the estimation accuracy.

Digital image analysis appears to be more and more relevant to the study of physical phenomena involving fluid motion, and of their evolution over time. In that context, 2D deformable motion analysis is one of the important issues to be investigated. The interpretation of such deformable 2D flow fields can generally be stated as the characterization of linear models provided that first order approximations are considered in an adequate neighborhood of so-called singular points, where the velocity becomes null. This paper describes an efficient method, based on a statistical approach, which explicitly addresses these problems, and allows us to locate, characterize and track such singular points in an image sequence. It does not require the prior computation of the velocity field. The method has been validated by experiments carried out with synthetic and real examples corresponding to meteorological image sequences. In fact, the described approach can be of interest in different applications dealing with the characterization of vector fields.

This paper deals with motion-segmentation, that is, with the partitioning of the image into regions of homogeneous motion. Here, homogeneous means that in each region a 2D polynomial model (e.g. an affine one) is able to describe at each location the underlying "true" motion with a predefined precision /spl eta/. However, no estimation of this true motion field is required. The motion models are computed using a multiresolution robust estimator. Therefore, as opposed to almost all other motion-segmentation scheme, the motion model of a given region only needs to be estimated once at a given time instant. Moreover, the determination of the boundaries between the different regions, which is stated as a statistical regularization based on a multiscale Markov random field (MRF) modeling, only requires one pass. Finally, thanks to the definition of an explicit detection step of areas where the error between the underlying motion and the one given by the estimated models is not within the precision /spl eta/, we are able to get a good segmentation from the very beginning of the sequence, and to manage the appearance of new objects in the scene, as well as the momentary increase in the complexity of motion in already existing regions. Results obtained on many real image sequences have validated our approach.

This paper is concerned with the problem of computing normal displacements along contours in image sequences. Our estimation is restricted to the perpendicular-to-the-edge velocity component, since the well-known "aperture problem" restricts any local estimation to this only component. We model moving edges as spatio-temporal surface patches in the image sequence space (x, y, t). A statistical regularization scheme based on Markov random fields allows us to get a homogeneous and relevant normal motion field along contours. It turns out that it can be implemented in an efficient way, mostly leading to convolution-like computations. Subpixel accuracy comes straightforwardly with this modeling, and is handled within the optimization stage itself, not as a post-processing step. Results are presented concerning synthetic experiments and real-world sequences.

This paper deals with the recovery of 3D information using a single mobile camera in the context of active vision. We propose a general revisited formulation of the structure-from-motion issue, and we determine adequate camera configurations and motions which lead to a robust and accurate estimation of the 3D structure parameters. We apply the visual servoing approach to perform these camera motions. Real-time experiments dealing with the 3D structure estimation of points and cylinders are reported, and demonstrate that this active vision strategy can very significantly improve the estimation accuracy.>

Locating the image of moving objects in a scene and estimating their motion are among the main issues in dynamic scene analysis. They are relevant to a large variety of tasks such as autonomous navigation, tracking, obstacle detection and surveillance. A central problem faced by existing methods is related to the occurrence of motion boundaries. With the aim of achieving correct and robust interpretation in the face of occurrence of motion boundaries, noise, and insufficiently informative image brightness pattern regions, the method we propose is articulated about the fundamental concepts of gradient-based multiconstraint, statistically robust regression, and regularization. >

We address the problem of detecting moving objects from a moving camera. The apparent flow field induced by the camera motion is modeled by a 2D parametric motion model and compensated for using the values of the parameters estimated by a multiresolution robust method. Motion detection is achieved through a statistical regularization approach based on multiscale Markov random field (MRF) models. Particular attention has been paid to the definition of the energy function involved and to the considered observations. This method has been validated by experiments carried out on different real image sequences.>

This paper describes a new approach to track complex primitives along image sequences - integrating snake-based contour tracking and region-based motion analysis. First, a snake tracks the region outline and performs segmentation. Then the motion of the extracted region is estimated by a dense analysis of the apparent motion over the region, using spatio-temporal image gradients. Finally, this motion measurement is filtered to predict the region location in the next frame, and thus to guide (i.e. to initialize) the tracking snake in the next frame. Therefore, these two approaches collaborate and exchange information to overcome the limitations of each of them. The method is illustrated by experimental results on real images.

In this paper, we present a method to detect moving objects in a scene modelled as a set of line segments and using the measure of their 2D motion through a monocular sequence of images. These measures are supplied by a segment tracker using a Kalman filtering approach. In order to perform such a detection task, we have to determine and sort the rigid structures; we work with structures of five segments and declare that they are rigid if they verify several rigidity conditions. Using these constraints, we sort them into "classes" according to their associated 2D affine motion fields.

The problem of establishing trajectories of objects in a long image sequence is addressed, in the case of occlusion, disocclusion of objects, and crossing trajectories and junctions. Two complementary criteria are investigated in order to arrive at the decision of linking two partial pieces of trajectory which could come from a single object in motion, i.e., the continuity of the global trajectory, and the continuity of the velocity of the moving object. Experiments are conducted on long sequences of real images. Complete trajectories are successfully recovered.>

The authors present a scheme for motion detection exploiting temporal integration and local contextual information. A multiscale temporal decomposition is supplied to the original sequence. Change detection is performed using a likelihood test at each temporal scale. The decision process is formalized within a statistical regularization framework and takes advantage of a tracking module. Motion detection is achieved by minimizing an energy function. This function involves three terms, expressing (1) adequacy between temporal variations at different scales and motion labels, (2) local spatial regularization, and (3) coherence between temporal prediction of change area locations and motion labels. Experimental results on real scenes are reported.>

The estimation of dense velocity fields from image sequences is basically an ill-posed problem, primarily because the data only partially constrain the solution. It is rendered especially difficult by the presence of motion boundaries and occlusion regions which are not taken into account by standard regularization approaches. In this paper, the authors present a multimodal approach to the problem of motion estimation in which the computation of visual motion is based on several complementary constraints. It is shown that multiple constraints can provide more accurate flow estimation in a wide range of circumstances. The theoretical framework relies on Bayesian estimation associated with global statistical models, namely, Markov random fields. The constraints introduced here aim to address the following issues: optical flow estimation while preserving motion boundaries, processing of occlusion regions, fusion between gradient and feature-based motion constraint equations. Deterministic relaxation algorithms are used to merge information and to provide a solution to the maximum a posteriori estimation of the unknown dense motion field. The algorithm is well suited to a multiresolution implementation which brings an appreciable speed-up as well as a significant improvement of estimation when large displacements are present in the scene. Experiments on synthetic and real world image sequences are reported.>

Presents a motion detection method based on a temporal multiscale transform, able to determine the different dynamic components. The authors consider a sequence of images as a set of monodimensional temporal signals, decomposed on a suitable orthogonal basis of wavelets. In order to detect changes at various frequency levels, the authors introduce a local spatial interaction, by applying a likelihood test within small windows. Then the authors combine these change binary maps corresponding to the decomposition levels into a decision process to separate the moving objects from parasitical motion and noise. Experimental results on real scenes are presented.>

Addresses the problem of estimating time-to-collision maps involving all the objects in relative motion with respect to the camera. The approach only takes into account normal flows. Moreover the authors prove that first-order visual motion models are sufficient to obtain time-to-collision. Experiments have been carried out on real images to validate the performance of the method.>

The authors deal with analysis of the dynamic content of a scene from an image sequence whatever the camera situation is (static or mobile). This problem involves a motion-based segmentation step. A method which ensures stable motion-based partitions owing to a statistical regularization approach is presented. It does not require the explicit estimation of optic flow fields and manages to link those partitions in time. The identification of the kinematical components of the scene relies on an intermediate layer accomplishing a generic qualitative motion labeling. This is achieved considering jointly several successive images. No 3-D measurements are required. Results obtained on several real image sequences corresponding to complex outdoor situations are reported.>

An approach to the problem of optic flow estimation and segmentation from image sequences is presented. It is shown that optic flow estimation and segmentation can be expressed, within a Bayesian decision framework, as a global estimation problem. The unknown process to be estimated corresponds to the 2D relative velocity field and to the motion boundaries. Several observations are used in the scheme, involving the spatiotemporal gradients of the image sequence and the output of an intensity edge detector. The unknown velocity field and motion discontinuities are modeled using a joint Markov random field, allowing the smoothing of the velocity field and the preservation of motion boundaries. Critical areas, such as occluding regions, are detected using a likelihood test and, in this case, a modified interaction model is applied. Results are presented on a real-world digital TV sequence involving complex 3D motions and occlusions.>

A multimodal approach to the problem of velocity estimation is presented. It combines the advantages of the feature-based and gradient-based methods by making them cooperate in a single global motion estimator. The theoretical framework is based on global Bayesian decision associated with Markov random field models. The proposed approach addresses, in parallel, the problem of velocity estimation and segmentation. Results on synthetic as well as on real-world image sequences are presented. Accurate motion measurement and detection of motion discontinuities with a surprisingly good quality have been obtained.>

The authors address the problem of motion detection in an image sequence from the variations in time of the intensity distribution. The problem is not limited to change detection but encompasses the recovery of the projections of moving areas in the image. The approach is characterized by the joint treatment of the detection of temporal changes and the reconstruction of mobile object masks according to a probabilistic formulation. More formally, spatio-temporal contextual information is introduced through Markovian models, using Gibbs distributions defined on a spatio-temporal neighborhood system. Then the problem at hand is stated as a statistical labeling one. To decide whether or not a point belongs to a moving area is equivalent to assigning to it a given label. A solution to this labeling problem is formulated according to the maximum a posteriori (MAP) criterion. Experiments with a real image sequence have been carried out.>

No abstract available.

The determination of moving edges in an image sequence is discussed. An approach is proposed that relies on modeling principles and likely hypothesis testing techniques. A spatiotemporal edge in an image sequence is modeled as a surface patch in a 3-D spatiotemporal space. A likelihood ratio test enables its detection as well as simultaneous estimation of its related attributes. It is shown that the computation of this test leads to convolving the image sequence with a set of predetermined masks. The emphasis is on a restricted but widely relevant and useful case of surface patch, namely the planar one. In addition, an implementation of the procedure whose computation cost is merely equivalent to a spatial gradient operator is presented. This method can be of interest for motion-analysis schemes, not only for supplying spatiotemporal segmentation, but also for extracting local motion information. Moreover, it can cope with occlusion contours and important displacement magnitude. Experiments have been carried out with both synthetic and real images.>

This paper describes a model-based approach to perform tracking of extratropical atmospheric disturbances from a sequence of satellite cloud-cover images. More precisely, it deals with the estimation of motion of these spiral-shaped cloud systems (both translational and rotational motion), and the measurement of the evolution of their shape. Tracking is achieved by recording from one image to the next the changes of the model parameter values. A maximum likelihood criterion is used in the process of fitting model to sensed data. The defined model takes into account geometric and intensity aspects. Such an approach readily yields global information on the disturbance cloud system of interest. As a requirement in such an application is robustness to noise, to this end two versions of the modeling have been considered.