We propose a technique for the real-time synthesis of temporally interpolated video frames suitable for low bandwidth applications. The system is based on the warping of edge preserving meshes. Node selection segments the scene and places mesh nodes on top of object edges in the reference frame. Constrained Delaunay triangulation is used to create a mesh that keeps triangulation edges aligned with object edges in the scene. Node displacements are estimated and several interpolated frames may be synthesized between each pair of original frames without the need for fitting a new mesh to the reference frame. Each of the interpolated frames is synthesized by warping the mesh. We present experimental results that demonstrate the performance and applicability of the proposed method.

Very low bit-rate coding requires new paradigms that go well beyond pixel- and frame-based video representations. We introduce a novel content-based video representation using tridimensional entities: textured object models and pose estimates. The multiproperty object models carry stochastic information about the shape and texture of each object present in the scene. The pose estimates define the position and orientation of the objects for each frame. This representation is compact. It provides alternative means for handling video by manipulating and compositing three-dimensional (3-D) entities. We call this representation tridimensional video compositing, or 3DVC for short. We present the 3DVC framework and describe the methods used to construct incrementally the object models and the pose estimates from unregistered noisy depth and texture measurements. We also describe a method for video frame reconstruction based on 3-D scene assembly, and discuss potential applications of 3DVC to video coding and content-based handling. 3DVC assumes that the objects in the scene are rigid and segmented. By assuming segmentation, we do not address the difficult questions of nonrigid segmentation and multiple object segmentation. In our experiments, segmentation is obtained via depth thresholding. It is important to notice that 3DVC is independent of the segmentation technique adopted. Experimental results with synthetic and real video sequences where compression ratios in the range of 1:150-1:2700 are achieved demonstrate the applicability of the proposed representation to very low bit-rate coding.

We present an encoder rate control algorithm based on a perceptual criterion for very low bit rate video. The emerging H.263 recommendation for transmission of video over narrow bandwidth channels supports variable frame rate video streams. This additional freedom enables the rate control algorithm to explore the tradeoff between temporal sampling and spatial quality, while keeping the generated bit rate within the available channel bandwidth. The proposed criterion takes advantage of this tradeoff by using dynamic models of encoder performance and characteristics of the human visual system to adaptively set the nominal quantizer scale factor and instantaneous frame rate. Experiments with typical video-conference sequences demonstrate the application of this new algorithm to both constant and variable bit rate channels.

In order to achieve good quality very low bit rate video coding, new techniques leading to highly compact representations for video sequences must be investigated. We present a novel video codec framework, where video sequences are represented in terms of stochastic nonparametric 3-D object models and motion script estimates. Multi-property object models, carrying both shape and color information, are incrementally built from video and range sequences. Motion estimates are obtained by depth map registration. We refer to this framework as 3-D video compositing, or 3DVC for short. In this paper, we will describe 3DVC in detail, and present experimental results where interframe compression ratios in the range of 10/sup 2/ to 10/sup 3/ have been achieved.