We conceive the problem of multiple semantic video object (SVO) extraction as an issue of designing extensive operators on a complete lattice of partitions. As a result, we propose a framework based on spatial partition generation and application of optimal operators on the generated partitions. Based on a statistical analysis of the watershed algorithm, we develop a multivalued morphological spatial segmentation method that incorporates an edge-driven marker extraction algorithm and a growing method which integrates both color and edge information. Having embedded the problem in the partition lattice framework, we propose a spatio-temporal regional maximum likelihood operator for extraction purposes. Some theoretical properties of the operator are established. Experimental results on several MPEG-4 test video sequences show that our scheme improves the precision of the extracted SVO boundaries compared to traditional watershed algorithms and provides accurate tracking of multiple SVOs in both static and moving camera scenarios. Furthermore, this scheme can be extended to deal with more general interactive video authoring systems.

Occlusion/disocclusion is one of the fundamental problems for semantic video object (SVO) extraction, where pixel-wise accuracy is required. This issue is critical because the degradation in tracking due to object occlusion/disocclusion significantly increases the amount of user interaction required in off-line video editing applications. We present an approach based on the application of an extensive operator on a lattice of partitions, which exploits information from various views of the scene, based on a probabilistic formulation. Our multiview operator builds on the regional application of the maximum a posteriori principle, by integrating a single-view region classification stage with a multiview stage that improves classification for those disoccluded regions labeled as uncertain. Results on several real sequences show that our approach improves the SVO tracking compared to the single-view case and that, as a result, increases the quality of the extracted SVOs and reduces the total amount of user interaction.

The relation between morphological gray-level connected operators and segmentation algorithms based on region merging/classification strategies has been pointed out several times in the literature. However, to the best of our knowledge, the formal relation between them has not been established. This paper presents the link between the two domains based on the observation that both connected operators and segmentation algorithms share a key mechanism: they simultaneously operate on images and on partitions, and therefore they can be described as operations on a joint image-partition model. As a result, we analyze both segmentation algorithms and connected operators by defining operators on complete product lattices, that explicitly model gray-level and partition attributes. In the first place, starting with a complete lattice of partitions, we initially define the concept of the segmentation model as a mapping in a product lattice, whose elements are three-tuples consisting of a partition, an image that models the partition attributes, and an image that represents the gray-level model associated to the segmentation. Then, assuming a conditional ordering relation, we show that any region merging/classification segmentation algorithm can be defined as an extensive operator in such a complete product lattice, in the second place, we proposed a very similar lattice-based extended representation of gray-level functions in the context of connected operators, that highlights the mathematical analogy with segmentation algorithms, but in which the ordering relation is different. We use this framework to show that every region merging/classification segmentation algorithm indeed corresponds to a connected operator. While this result provides an explanation to previous work in the area, it also opens possibilities for further analysis in the two domains. From this perspective, we additionally study some theoretical properties of a general region merging segmentation algorithm.

Occlusion/disocclusion is one of the fundamental problems for semantic video object (SVO) extraction because the tracking degradation from occlusion/disocclusion creates difficulty for the pixel-wise accuracy requirement and adds to the amount of user interaction required in off-line video editing applications. We present an approach based on the application of a new extensive operator in a lattice of partitions, that exploits information from various views of the scene to address the disocclusion problem by using a regional Bayesian formulation. Our multiview operator builds on the application of the maximum a posteriori (MAP) principle, by integrating a single view region classification stage and a multiview stage that solves for disoccluded regions labeled as uncertain. Results on several real sequences show that this approach improves the quality of the extracted SVOs and reduces the total amount of user interaction.

We conceive the problem of multiple semantic wide object (SVO) extraction as an issue of designing extensive operators on the lattice of partitions. As a result, we propose a framework based on spatial partition generation and application of optimal operators on the generated partitions. The first stage is obtained with a multivalued morphological spatial segmentation method that incorporates an edge-driven marker extraction algorithm and a growing method which integrates both color and edge information. For the second stage, we propose a new spatio-temporal regional maximum likelihood partition operator for extraction purposes. Subjective and objective evaluation of the experimental results obtained with our approach on several MPEG-4 test video sequences show that it accurately tracks multiple SVOs in several scenarios, while improving the SVO extraction precision compared to traditional watershed techniques.

We present a novel algorithm for semantic video object (SVO) extraction, based on a new multivalued morphological spatial segmentation that integrates color and edge information, and object tracking by backward region classification. Our proposed spatial segmentation incorporates a new marker extraction method based on intensity edge information that improves the definition of the real borders of the scene objects, and a new distance criterion based on color and edge information to guide the watershed algorithm. Experimental results on several MPEG-4 test video sequences show that our algorithm improves the precision of the extracted SVO boundaries compared to the traditional water-shed technique, and that it is capable of tracking multiple SVOs in static and moving camera scenarios.

This paper introduces a novel semantic video object extraction system using mathematical morphology and a perspective motion model. Inspired by the results from the study of the human visual system, we intend to solve the semantic video object extraction problem in two separate steps: supervised I-frame segmentation, and unsupervised P-frame tracking. First, the precise semantic video object boundary can be found using a combination of human assistance and a morphological segmentation tool. Second, the semantic video objects in the remaining frames are obtained using global perspective motion estimation and compensation of the previous semantic video object plus boundary refinement as used for I frames.

This paper introduces a new technique in block-based motion estimation and motion compensation for object-based video coding. Out of many approaches for object-based coding, hybrid algorithms that take advantages of the traditional motion compensated block-based DCT approach are noted for their maturity and simplicity. Indeed, it has become the main feature of the upcoming international video standard MPEG-4. In this type of hybrid algorithm, the boundary of the objects calls for special attention. The discontinuity at object boundary could be very costly if not handled properly. We discovered that efficient motion estimation and compensation for the object boundary is most critical to the performance of object-based video coding. Our studies show that padding, a process of boundary value extension, is essential in motion estimation/compensation of arbitrarily shaped objects. We have designed novel padding algorithms and report our preliminary results in this paper.

This paper introduces a novel semantic video object extraction system using mathematical morphology and a perspective motion model. Inspired by the results from a study of the human visual system, we solve the semantic video object extraction problem in two steps: semi-automatic I-frame segmentation and automatic P-frame tracking. First, the precise semantic video object boundary can be found with the combined power of human assistance and a morphological segmentation tool. Second, the semantic video object in the remaining frames can be obtained using a global perspective motion estimation and compensation of the previous semantic video object plus boundary refinement as used for frames.

A layered video object coding system is presented in this paper. The goal is to improve video coding efficiency by exploiting the layering of video and to support content-based functionality. These two objectives are accomplished using a sprite technique and an affine motion model on a per-object basis. Several novel algorithms have been developed for mask processing and coding, trajectory coding, sprite accretion and coding, locally affine motion compensation, error signal suppression, and image padding. Compared with conventional frame-based coding methods, better experimental results on both hybrid and natural scenes have been obtained using our coding scheme. We also demonstrate content-based functionality which can be easily achieved in our system.

This paper presents a region-based coding algorithm for video sequences. The coding approach involves a time-recursive segmentation relying on the pixels homogeneity, a region-based motion estimation, and motion compensated contour and texture coding. This algorithm is mainly devoted to very low bit rate video coding applications. One of the important features of the approach is that no assumption is made about the sequence content. Moreover the algorithm structure leads to a scalable coding process giving various levels of quality and bit rates. The coding as well as the segmentation are controlled to regulate the bit stream. Finally, the interest of morphological tools in the content of region-based coding is extensively reviewed.>

An information-lossy method for contour image sequence coding is described in this paper. It is based on motion estimation, compensation and morphological filters. In order to exploit the high temporal redundancy of the contour image sequence, motion estimation and compensation are used for the correspondent label image sequence. The first frame is directly coded by the chain-code method. All the following frames are coded in motion compensation mode. Within each frame, labels are divided into foreground labels and background labels. Only the prediction error for the foreground labels are coded. Morphological filters are applied to clean the prediction error for the foreground labels. Afterwards, an efficient prediction error coding algorithm is introduced based on the modified chain-code method. Experimental results have demonstrated outstanding performance for the coding of a contour image sequence.>

In order to solve the bottleneck problem of region-based coding scheme, which is the contour coding, lossy methods are introduced in this paper for both 2D and 3D (2D plus time) contour image(s). A non-linear filter by means of majority operations is designed to act as contour simplification for both pre-processing and post-processing of contour coding. A 20% gain in contour coding is achieved by applying this filter while the contour location error is within one pixel width. Afterwards, an efficient contour motion estimation and compensation coding method is presented. A fast full search motion estimation algorithm is also described, which is 2R times faster than traditional method (R is the region number). Experimental results have demonstrated outstanding performance for very low bit rate video coding.>

This paper describes a segmentation-based coding algorithm for very low bit rate. The algorithm involves a segmentation relying on a time recursive morphological segmentation technique. It extracts both large and contrasted regions. Then, a simple region-based motion estimation is performed. Finally, the sequence is coded by contour-texture motion compensation. The proposed method leads to very low bit rates while being able to represent the meaningful regions of the scene. Moreover, it makes no assumption about the scene content and can be used for any sequence.

A novel method for stereo correspondence that makes use of integrating multiresolution and hierarchical primitives is proposed. The pyramid is used for multiresolution. For each resolution level the hierarchy of primitives is constructed: zero crossing points, their straight line segment, and structure points. The hierarchy of the lower level is matched under the guidance of the hierarchy of the higher level, except the top level. The same is true for the matching of the hierarchy in the same resolution level. Experiments were carried out for a variety of stereo images, showing that the proposed method is effective, efficient, and robust.>