Scalability is an attractive solution to the problems posed by modern multimedia transmission systems. The key for better performance in scalable coding is joint rate distortion optimization. We present a new stochastic model for scalable coding which models the dependencies among layers, and we derive an optimal joint quantizer selection scheme for memoryless Gaussian sources. Simulation results verify the validity of the theoretical analysis.

Despite the attention that hierarchical B picture coding has received, little attention has been given to a related technique called hierarchical P picture coding. P picture only coding without reverse prediction is necessary in constrained bit rate applications that require ultra-low delay and/or low complexity, such as videoconferencing. Such systems, however, have been using the traditional IPPP picture coding structure almost exclusively. In this paper, we investigate the use of hierarchical P coding vs. traditional IPPP coding and demonstrate that it has significant advantages which have not yet been well documented or understood. From a pure coding efficiency point of view, we show that for encoders configured to use ultra-low delay and low complexity coding tools, hierarchical P coding achieves an average advantage of 7.86% BD-rate and 0.34 dB BD-SNR.

We propose a highly efficient, look-up table-based, renormalization method that can be used by any binary arithmetic encoder with the follow-on procedure. It replaces the time- consuming branching operations in the renormalization process with table look-ups and some simple bit-wise operations. We show that our new renormalization method outperforms the currently used ones with much less memory requirement than the previously known, table-based quasi-coder.

Scalable video coding (SVC) and transmission has been a research topic for many years. Among other objectives, it aims to support different receiving devices, perhaps connected through a heterogeneous network structure, using a single bit stream. Earlier attempts of standardized scalable video coding, for example in MPEG-2, H.263, or MPEG-4 Visual, have not been commercially successful. Nevertheless, the Joint Video Team has recently focused on the development of the scalable video extensions of H.264/AVC, known as SVC. Some of the key problems of older scalable compression techniques have been solved in SVC and, at the same time, new and compelling use cases for SVC have been identified. While it is certainly important to develop coding tools targeted at high coding efficiency, the design of the features of the interface between the core coding technologies and the system and transport are also of vital importance for the success of SVC. Only through this interface, and novel mechanisms defined therein, applications can take advantage of the scalability features of the coded video signal. This paper provides an overview of the system interface features defined in the SVC specification. We discuss, amongst other features, bit stream structure, extended network abstraction layer (NAL) unit header, and supplemental enhancement information (SEI) messages related to scalability information.

Motion vector field analysis in motion-assisted rate control (MARC) helps us interpret the relationship between a distribution of motion vectors and its visual characteristics. In our earlier work, we showed how MARC utilized this relationship into an effective rate control method that enhanced subjective quality of regions with high visual sensitivity. In this work, we extend our initial idea further to enhance its performance. First, visual features of interest are defined in the sense of local distribution of pre-filtered motion vectors and their relationship with human visual sensitivity is discussed. To estimate the degree of this sensitivity, two simple and effective motion metrics are designed and their performance is evaluated on an MARC-enabled H.264 video codec. The calculation overhead is minimal, since motion vector information is already provided by the H.264 video codec. Our result shows that MARC reduces temporal PSNR fluctuations across frames while it improves the subjective quality of visually sensitive regions.

We discuss new theoretical and experimental results on the dynamic rate shaping (DRS) approach for transcoding compressed video bitstreams (MPEG-1, MPEG-2, MPEG-4, H.261, as well as JPEG). We analyze the behavior of DRS assuming a first order autoregressive source. We propose a set of low complexity algorithms for both constrained and unconstrained DRS and substantiate the almost-optimal experimental performance of the memoryless algorithm by assuming a first order autoregressive source. By deriving the statistical and rate-distortion characteristics of different components of the interframe rate shaping problem, we offer an explanation as to why the set of optimal breakpoint values for any frame is somewhat invariant to the accumulated motion compensated shaping error from past frames. We also present an extensive experimental study on the various DRS algorithms (causally optimal, memoryless, and rate-based) both in their constrained and generalized forms. The study proves the computational viability of the DRS approach to transcoding and identifies a range of rate shaping ratios for which it is better than requantization, both complexity-wise as well as in performance. This result is significant in that it opens up the way to construct much simpler memoryless algorithms that give minimal penalty in achieved quality, not just for this but possibly other types of algorithms. This is also the very first use of matrix perturbation theory for tracking the spectral behavior of the autocorrelation matrix of the source signal and the motion residual it yields.

We introduce a video newsgathering system called mobile broadcast video. The system is designed to use a wireless LAN and low earth orbit satellite technology for increased mobility and wide area coverage. It offers broadcast-quality video by using the MPEG-2 standard for audio/video compression and multiplexing. Our prototype uses off-the-shelf hardware components with all system software developed in-house. We present our implementation, focusing particularly on real-time live video acquisition and fast delivery of the high-rate compressed data over a wireless LAN. We also introduce an effective MPEG-2 transport stream level error concealment scheme that increases subjective played-back audiovisual quality in the case of packet losses. Through experimental results, we demonstrate that an economical, high-quality, real-time solution for mobile video newsgathering and broadcasting is realizable using even wireless 802.11b LANs.

A highly efficient, lookup table-based, renormalization method is proposed, which can be used by any binary arithmetic coder with the follow-on procedure. It replaces the time-consuming branching operations in the renormalization process with table lookups and some simple bit-wise operations. We show that the proposed method is much faster than the one used by the M coder in the CABAC entropy coding scheme of the H.264 video coding standard. Our method requires much less memory than the previously known, table-based quasi-coder.

We propose the concept of spatio-temporal model-assisted compatible coding, a technique that selectively encodes areas of different importance to the human eye in terms of space and time in moving images. Our motivation is the fact that eye contact and lip synchronization are very important in person-to-person communication. Several areas including the eyes and lips require different types of quality since they have different perceptual significance to human observers. For example, for the eyes, high resolution is needed for clear communication, while for the lips frequent refresh is desirable. Fully automatic algorithms are described for performing model area detection and motion tracking. An area-selective (AS) rate control scheme is proposed, in order to effect AS bit allocation. It extends prior work that dealt with only the spatial dimensions to include the temporal aspects of the video signal as well. The decoder does not need to be changed in any way although the encoder's rate control unit is modified. As a result, the scheme can be used in compliance with standards-based codecs. As the scene model is only used in the rate control module (hence the term "model-assisted"), the technique does not suffer from the artifacts associated with model-based coding. Extensive experimental results are provided using ITU-T H.263, addressing very low bit-rate compression (below 64 kb/s).

We present Flavor, a formal language for audio-visual object representation, that has been developed to describe any coded multimedia data in formats such as GIF, JPEG and MPEG. The language comes with a translator for generating C++/Java code from the Flavor description, and the generated code can include bitstream reading, writing and tracing methods. Since Version 5.0, the translator has been enhanced to also support XML. With the enhanced translator, the generated C++/Java code can include a method for producing XML documents that correspond to the bitstreams described by Flavor. The description can also be used to generate a corresponding XML schema. Additionally, a software tool for converting XML representation of multimedia data back into bitstream form is provided.

We analyze the problem of optimal data partitioning of MPEG-2 coded video in an operational rate-distortion context. The optimal algorithm is characterized and shown to have high complexity and delay. A causally optimal algorithm based on Lagrangian optimization is proposed, that optimally solves the problem for intra (I) pictures, and provides an optimal solution for predicted/interpolated (P/B) pictures when the additional constraints of causal operation and/or low-delay are imposed. A memoryless version of the algorithm, theoretically optimal for intrapictures only, is shown to perform almost identically but with significantly less computational complexity. Finally, a fast, suboptimal algorithm using purely rate-based optimization is also proposed, and is shown to perform quite close (within 1 dB) to the causally optimal one. Experimental results are provided using actual MPEG-2 coded video bitstreams.

Object-based representation of audio-visual (AV) presentations provides a flexible scheme to create interactive content that lends itself to resource-driven adaptation. The content adaptation needs of mobile devices can be met well with the use of object-based AV presentations. The main distinguishing feature of object-based AV presentations is the scene composition at the user terminal. In this paper, we discuss the problem of scheduling the delivery of object-based AV presentations under resource constraints. We explore the similarities with the problem of job sequencing on a single machine. We present a family of algorithms to determine the schedulability of AV presentations, and for unschedulable presentations, we present algorithms to compute a schedule that minimizes the additionally acquired resources. We present algorithms for computing incremental schedules for applications such as content authoring that require immediate feedback on resource consumption. The algorithms can be used to schedule object-based MPEG-4 presentations. We discuss the algorithms and results by considering a relatively complex MPEG-4 presentation with 16 objects, including audio, video, and images.

We discuss the behavior of the optimal solution to the dynamic rate shaping problem assuming an AR(1) source model. By analyzing the statistical and rate-distortion behavior of the different components of this minimization problem, the following key result is mathematically proven: "the set of optimal breakpoint values for any frame is invariant to the accumulated motion compensated shaping error from past frames and may be very reasonably approximated using the current frame shaping error alone".

Complexity distortion theory (CDT) is a mathematical framework providing a unifying perspective on media representation. The key component of this theory is the substitution of the decoder in Shannon's classical communication model with a universal Turing machine. Using this model, the mathematical framework for examining the efficiency of coding schemes is the algorithmic or Kolmogorov (1965) complexity. CDT extends this framework to include distortion by defining the complexity distortion function. We show that despite their different natures, CDT and rate distortion theory (RDT) predict asymptotically the same results, under stationary and ergodic assumptions. This closes the circle of representation models, from probabilistic models of information proposed by Shannon in information and rate distortion theories, to deterministic algorithmic models, proposed by Kolmogorov in Kolmogorov complexity theory and its extension to lossy source coding, CDT.

A statistical model-based video segmentation algorithm is presented for head-and-shoulder type video. This algorithm uses domain knowledge by abstracting the head-and-shoulder object with a blob-based statistical region model and a shape model. The object segmentation problem is then converted into a model detection and tracking problem. At the system level, a hierarchical structure is designed and spatial and temporal filters are used to improve segmentation quality. This algorithm runs in real time over a QCIF size video, and segments it into background, head and shoulder three video objects on average Pentium PC platforms. Due to its real time feature, this algorithm is appropriate for real time multimedia services such as videophone and web chat. Simulation results are offered to compare MPEG-4 performance with H.263 on segmented video objects with respects to compression efficiency, bit rate adaptation and functionality.

We discuss formulations to solve the bit-allocation problem with dependencies. These formulations are more generic than those known earlier in that we relax a so-called causality assumption. We study the incremental improvement obtained by having a dependency structure in optimal quantization choices. As a way to solve this problem approximately, we discuss novel rounding procedures based on some relaxation of the original problem (either the LP relaxation or a Lagrange relaxation). The rounding algorithms perform extremely close to optimality. Our results also show an improvement of around 0.5 dB over the best causally-optimal solution.

We present a novel interactive object segmentation algorithm: Rubberband. Rubberband is motivated by graph search based boundary detection algorithms but has greatly improved robustness and controllability performance with (1) a novel user interface; (2) an improved graph search algorithm; (3) an automatic scale selection algorithm.

Flavor is a formal language developed for describing compressed bitstream syntax, especially the bitstreams representing multimedia objects. A translator is also available for translating a Flavor description into C++ or Java code. The code includes bitstream parsing and generating operations that can readily be used by software developers with essentially zero additional programming. After briefly describing the latest version of Flavor, we propose an extension - XFlavor - that provides XML features by transforming the traditional bitstream representation of multimedia data into a corresponding XML representation.

We propose an optimal buffered compression algorithm for shape coding as defined in the forthcoming MPEG-4 international standard. The MPEG-4 shape coding scheme consists of two steps: first, distortion is introduced by down and up scaling; then, context-based arithmetic encoding is applied. Since arithmetic coding is "lossless," the down up scaling step is considered as a virtual quantizer. We first formulate the buffer-constrained adaptive quantization problem for shape coding, and then propose an algorithm for the optimal solution under buffer constraints. Previously, the fact that a conversion ratio (CR) of 1/4 makes a coded image irritating to human observers for QCIF size was reported for MPEG-4 shape coding. Therefore, careful consideration for small size images such as QCIF should be given to prevent coded images from being unacceptable. To this end, a low bit rate tuned algorithm is proposed in this paper as well. Experimental results are given using an MPEG-4 shape codec.

We propose a fast face detection algorithm that works directly on the compressed DCT domain. Unlike the previous DCT domain processing designs that are mainly based on skin-color detection, our algorithm analyzes both color and texture information contained in the DCT parameters, therefore could generate more reliable detection results. Our texture analysis is mainly based on statistical model training and detection. A number of fundamental problems, e.g., block quantization, preprocessing in the DCT domain, and feature vector selection and classification in the DCT domain, are discussed.

A novel and simple approach for defining end-to-end quality of service (QoS) in video-on-demand (VoD) services is presented. Using this approach, we derive a schedulable region for a video server which guarantees end-to-end QoS, where a specific QoS required in the video client translates into a QoS specification for the video server. Our methodology is based on a generic model for VoD services, which is extendible to any VoD system. In this kind of system, both the network and the video server are potential sources of QoS degradation. Specifically, we examine the effect that impairments in the video server and video client have on the video quality perceived by the end user. The Columbia VoD testbed is presented as an example to validate the model through experimental results. Our model can be connected to network QoS admission control models to create a unified approach for admission control of incoming video requests in the video server and network.

The extensive use of discrete transforms in image and video coding suggests the investigation on filtering before downsampling (FBDS) and filtering after upsampling (FAUS) methods directly acting on the transform domain. We describe the "transform-damain resolution translation" technique that gives flexibility to resize windows of each video conferencing session for a server compositing without explicit decompression, spatial domain processing, and compression. We generalize transform-domain filtering (TDF) to include nonuniform and multirate cases to implement the transform-domain resolution translator. The former is defined as a TDF problem in which the original transform-domain is of different size from the target one, while the latter considers the implementation of sampling rate conversion in the transform-domain. The implementation architecture is based on a pipeline that involves matrix-vector product blocks and vector addition, but is not limited to particular hardware. Such techniques are particularly useful for fast algorithms for processing compressed images and video where transform coding is extensively used (e.g., in JPEG, H.261, MPEG-1, MPEG-2, and H.363).

The current MPEG-4 shape coding scheme is "bitmap-based", and consists of two steps: first, distortion is introduced by down and up scaling; then, context-based arithmetic encoding is applied. Recently bitmap-based methods have been reported to produce more irritating images than contour-based methods, while introducing a similar amount of quantization error. In this paper, we propose four morphological enhancement methods which can be used in the current MPEG-4 shape decoding process. In addition, we devise a way to let decoders know which method is the best of those four morphological operations, thus preventing post-processing processes from being "blind" about the original source images. Experimental results are given using an MPEG-4 shape codec with morphological operations.

An active-contour interpolation approach is proposed for semi-automatic video object generation. In this approach, user can define the contour of the same video object on multiple frames and the computer interpolates the missing contours automatically. A typical active contour model is adapted and the contour interpolation problem is decomposed into two directional contour tracking problems and a merging problem. In addition, a new concept of parametric neighborhood template is introduced in order to improve the robustness of contour tracking.

MPEG-4, under the auspices of the ISO, is specifying tools to enable object-based audio-visual presentations. These include tools to encode individual objects, compose presentations with objects, store these object-based presentations and access these presentations in a distributed manner over networks. The main distinguishing feature of object-based audio-visual presentations is the scene composition at the user terminal. The objects that are part of a scene are composed and displayed at the user end as opposed to encoding the composed scenes as is done in the case of MPEG-2. Such object-based representation and presentation has several benefits including compression efficiency and interaction with individual objects.

An interactive authoring system is proposed for semi-automatic video object segmentation and annotation. This system features a new contour interpolation algorithm, which enables the user to define the contour of a video object on multiple frames while the computer interpolates the missing contours of this object on every frame automatically. Typical active contour (snake) model is adapted and the contour interpolation problem is decomposed into two directional contour tracking problems and a merging problem. In addition, new user interaction models are created for the user to interact with the computer. Experiments indicate that this system offers a good balance between algorithm complexity and user interaction efficiency.

We describe the implementation of a streaming client-server system for object-based audio-visual presentations in general and MPEG-4 content in particular. The system augments the MPEG-4 demonstration software implementation (IM1) for PCs by adding network-based operation with full support for the Delivery Multimedia Integration Framework (DMIF) specification, a streaming PC-based server with DMIF support (via Xbind Inc.'s XDMIF suite), and multiplexing software. We describe XDMIF, the first reference implementation of the DMIF specification. The MPEG-4 server is designed for delivering object-based audio-visual presentation. We discuss the issues in the design and implementation of MPEG-4 servers. The system also implements a novel architecture for client-server interaction in object-based audio-visual presentations, using the mechanism of command routes and command descriptors. This new concept of command routes and command descriptors is useful in developing sophisticated interactive applications.

In this paper, we derive an optimal rate control algorithm for shape coding as defined in the forthcoming MPEG-4 international standard. The current MPEG-4 shape coding scheme consists of two steps: first, distortion is introduced by down and up scaling; then, context-based arithmetic encoding is applied. Since arithmetic coding is "lossless", the down up scaling step is considered as a virtual quantizer. We first formulate the buffer-constrained adaptive quantization problem for shape coding, and then propose an algorithm for the optimal solution under buffer constraints. Experimental results are given using an MPEG-4 shape codec.

We describe the key features of the MPEG-4 systems specification and their encoding methodology: scene description and BIFS, animation streams, object descriptors, object content information, as well as multiplexing. We also describe our prototype software (Zest) for MPEG-4 authoring and playback.

Flavor (formal language for audio-visual object representation) is a new programming language for media-intensive applications. It extends the typing system of C++ and Java to incorporate bitstream representation semantics. This allows describing in a single place both the in-memory representation of data as well as their bitstream-level (compressed) representation as well. In this paper, we present our approach of hybrid variable length code segmentation in program code generation for parsing entropy-coded data from the bitstream. We also show that this approach produces a very good tradeoff between space (memory required) and time (decoding speed).

We present a new technique for streaming real time video on today's Internet, based ondynamic rate shapingandTCP congestion control. Dynamic rate shaping is a signal processing technique that adapts the rate of compressed video (MPEG-1, MPEG-2, H.26x) to dynamically varying bandwidth constraints. This provides an interface (or filter) between the source and the network, with which the encoder's output (either live or stored) can be perfectly matched to the network's available bandwidth. We couple this adaptation capability with the use of a new semi-reliable protocol that uses the TCP congestion window to pace the delivery of data into the network, but without using other TCP algorithms that are poorly suited to real time media. Use of TCP congestion control ensures that the protocol competes fairly with all other TCP data and that it optimally shares the available bandwidth. It also avoids the latency problems commonly associated with TCP. In addition, we describe a real application that uses this approach to stream MPEG video on the Internet. We present several experiments, performed in both a controlled environment and the wide area Internet, that were used to evaluate the effectiveness and fairness of the scheme. The results show that the proposed solution achieves superior video quality while at the same time providing fairness by sharing bandwidth equally with other non-real-time connections.

Content creation, editing, and searching are extremely time-consuming tasks that often require substantial training and experience, especially when high-quality audio and video are involved. New media represents a new paradigm for multimedia information representation and processing, in which the emphasis is placed on the actual content. It thus brings the tasks of content creation and searching much closer to actual users and enables them to be active producers of audio-visual information rather than passive recipients. We discuss the state of the art and present next-generation techniques for content representation, searching, creation and editing. We discuss our experiences in developing a Web-based distributed compressed video editing and searching system (WebClip), a media-representation language (Flavor) and an object-based video authoring system (Zest) based on it, and a large image/video search engine for the World Wide Web (WebSEEk). We also present a case study of new media applications based on specific planned multimedia education experiments with the above systems in several K-12 schools in Manhattan, NY.

We propose an architecture which can support video services in networks which have no quality of services (QoS) guarantees. We demonstrate that there is a need for such services today and also in the future, when QoS will be built into most networks. We then describe our current Internet-based system, which combines both image processing and networking techniques in order to provide good quality video without harming the performance of the Internet. These techniques are shown to be superior to previous work for the environment of video services operating in the Internet. We also present experimental results gathered in both a controlled environment and in external wide area connections across the Internet.

Sow and Eleftheriadis (see Proceedings of IEEE International Symposium on Information Theory, Germany, p.188, 1997) have introduced complexity distortion theory, a mathematical framework characterizing the design of programmable communication systems. In this paper, we show how complexity distortion theory fits in the MPEG-4 context and more generally in any system allowing programmability, by formalizing the concept of programmable decoders. We also show how it can be used to design intelligent encoders at two flexibility levels: the first one corresponding to the case where flexibility in the algorithm selection is allowed and the second where downloading of new tools for representation is also allowed.

We present the design and implementation of a new programming language for media-intensive applications called Flavor (Formal Language for Audio-Visual Object Representation). It is an extension of C++ and Java in which the typing system is extendedto incorporale bitstream representation semantics.This allows fo describe in a single place both the in-memory representation of data as well as their b&ream-level (compressed) representation as well.We have developed sofhvare tools (wnwee.columbia.edu&avor)that automatically generate standard C++ and Java code from the Flavor source code, so that direct access to compressed multimedia information by application developers can be achieved with essentially zero programming.

We propose an architecture which can support video services in networks which have no Quality of Services (QoS) guarantees. We demonstrate that there is a need for such services today and also in the future, when QoS will be built into most networks. We then describe our current Internet-based system, which combines both image processing and networking techniques in order to provide good quality video without harming the performance of the Internet. These techniques are shown to be superior to previous work for the environment of video services operating in the Internet.

The state of the standardization of the Systems part of ISO/IEC JTC1/SC29/WG11 (MPEG-4 Systems) as specified early 1997 is presented. First, a rationale and the architecture of the complete Systems are described. Based on the rapid technological advances in software and in hardware, the MPEG-4 Systems provides for a framework for the integration of natural and synthetic streamed and synchronised media. The different fields of interest in Systems are then developed from the description of audiovisual scenes to the definition of the multiplex. The programmability of the standard is described for composition and decompression and the language adopted by MPEG-4 on purpose of syntax description is fully detailed. Finally, a conclusion and the future evolutions of the specifications are presented.

We present the design of a novel syntactic and semantic framework to describe the bitstream-level representation of audio-visual information objects. This framework is important for emerging techniques in video compression, and is currently part of the MPEG-4 System Description Language Working Draft. The design is based on a principle of orthogonality between parsing and processing, i.e., separating the bitstream parsing from the other decoding operations, and extends the basic typing system of a programming language so that it incorporates sophisticated parsing information. Examples of object representations are provided using the MPEG-2 video bitstream syntax.

We introduce the concept of spatio-temporal model-assisted compatible (STMAC) coding, a technique to selectively encode areas of different importance to the human eye in terms of space and time in moving images. For this, we use the fact that human "eye contact" and "lip synchronization" are very important in person-to-person communication. Several areas including the eyes and lips need different types of quality, since different areas have different perceptual significance to human observers. For example, for the eyes "high resolution" is needed for clear communication, while for the lips "frequent refresh" is needed. The approach provides a better rate-distortion tradeoff than conventional image coding technologies based on MPEG-1, MPEG-2, H.261, as well as H.263, since STMAC coding is applied on top of an encoder, taking full advantage of its core design. The decoder does not need to be changed in any way although the encoder's rate control unit is slightly modified. This characteristic leads to the name "compatible" in the proposed concept. Experimental results are given using ITU-T H.263, addressing very low bit rate compression (13-17 Kbps).

This paper describes our progress in developing an advanced video-on-demand (VOD) testbed, which will accommodate various multimedia research and applications such as Electronic News on Demand, Columbia's Video Course Network, and Digital Libraries. Two different prototypes have been completed. The first generation of the testbed was based on a constant-bit-rate (CBR) video server utilizing Ethernet delivery. Contents were encoded and stored as MPEG-2 audio/video elementary streams. Software encoders/decoders were used in content generation and playback. The second generation of the testbed was enhanced with the capability of transmitting true MPEG-2 transport streams over the campus ATM network as well as the wide-area NYNET ATM network. A real-time video pump and a distributed application control protocol (MPEG-2's DSM-CC) have been incorporated. Hardware decoders and set-tops are being incorporated to test wide-area video interoperability. Our VOD testbed also provides an advanced platform for implementing proof-of-concept prototypes of related research. Our current research focus covers video transmission with heterogeneous quality-of-service (QoS) provision, video storage architecture design, content-based video indexing and browsing, multi-resolution (MR) video coding, efficient manipulation of compressed video, and advanced user interfaces. An important aim is to enhance interoperability. Accommodation of practical multimedia applications and interoperability testing with external VOD systems are currently being undertaken.

We introduce the concept of dynamic rate shaping, a technique to adapt the rate of compressed video bitstreams (MPEG-1, MPEG-2, H.261, as well as JPEG) to dynamically varying bit rate constraints. The approach provides an interface (or filter) between the encoder and the network, with which the encoder's output can be perfectly matched to the network's quality of service characteristics. Since the presented algorithms do not require interaction with the encoder, they are fully applicable to precoded, stored video (e.g., video-on-demand systems). By decoupling the encoder and the network, universal interoperability can be achieved. In essence, DRS bridges the gap between CBR and VBR video, allowing a continuum of possibilities between the two. A set of low-complexity algorithms for so-called unconstrained dynamic rate shaping are presented, and both optimal and extremely fast designs are discussed. Experimental results are provided using actual MPEG-2 bitstreams.

We describe a method of object-selective quantizer control in a standard coding system based on MC DCT-CCITT Recommendation H.261. The approach is based on two novel algorithms, namely buffer rate modulation and buffer size modulation. By forcing the rate control algorithm to transfer a relatively small fraction-about 10-15% on average-of the total available bit rate from the coding of the non-facial to that of the facial area, in head-and-shoulders videoteleconferencing sequences, images with better-rendered facial features are obtained, i.e. blocky artifacts in the facial area are less pronounced and eye contact and lip-sync are preserved. The improvement was found to be perceptually significant on video sequences coded at the rate of 64 kbps, with 48 kbps allocated for the input (color) video signal in QCIF format.

An architecture and associated protocols are presented for managing multicast addresses and performing connection control for multicast communication applications. A scheme to partition the multicast address space on the basis of the network number is proposed (an underlying IP-based internetworking environment is assumed), and its performance and scaling characteristics are discussed. A protocol is then developed to provide for dynamic allocation and release of multicast addresses, as well as maintaining state information for a connection. The protocol is independent of the address partitioning scheme; it is also shown to be robust and efficient. Finally, we describe two different mechanisms that enable the use of a common port number by all session participants.

We describe a distributed architecture for managing multicast addresses in the global Internet. A multicast address space partitioning scheme is proposed, based on the unicast host address and a per-host address management entity. By noting that port numbers are an integral part of end-to-end multicast addressing we present a single, unified solution to the two problems of dynamic multicast address management and port resolution. We then present a framework for the evaluation of multicast address management schemes, and use it to compare our design with three approaches, as well as a random allocation strategy. The criteria used for the evaluation are blocking probability and consistency, address acquisition delay, the load on address management entities, robustness against failures, and processing and communications overhead. With the distributed scheme the probability of blocking for address acquisition is reduced by several orders of magnitude, to insignificant levels, while consistency is maintained. At the same time, the address acquisition delay is reduced to a minimum by serving the request within the host itself. It is also shown that the scheme generates much less control traffic, is more robust against failures, and puts much less load on address management entities as compared with the other three schemes. The random allocation strategy is shown to be attractive primarily due to its simplicity, although it does have several drawbacks stemming from its lack of consistency (addresses may be allocated more than once).>

We present a novel and practical way to integrate techniques from computer vision to low bit-rate coding systems for video teleconferencing applications. Our focus is to locate and track the faces of persons in typical head-and-shoulders video sequences, and to exploit the face location information in a ‘classical’ video coding/decoding system. The motivation is to enable the system to selectively encode various image areas and to produce psychologically pleasing coded images where faces are sharper. We refer to this approach as model-assisted coding. We propose a totally automatic, low-complexity algorithm, which robustly performs face detection and tracking. A priori assumptions regarding sequence content are minimal and the algorithm operates accurately even in cases of partial occlusion by moving objects. Face location information is exploited by a low bit-rate 3D subband-based video coder which uses both a novel model-assisted pixel-based motion compensation scheme, as well as model-assisted dynamic bit allocation with object-selective quantization. By transferring a small fraction of the total available bit-rate from the non-facial to the facial area, the coder produces images with better-rendered facial features. The improvement was found to be perceptually significant on video sequences coded at 96 kbps for an input luminance signal in CIF format. The technique is applicable to any video coding scheme that allows for fine-grain quantizer selection (e.g. MPEG, H.261), and can maintain full decoder compatibility.

We present a novel and practical way to integrate techniques from computer vision to low bit-rate coding systems for video teleconferencing applications. Our focus is to locate and track the faces and selected facial features of persons in typical head-and-shoulders video sequences, and to exploit the location information in a ‘classical’ video coding/decoding system. The motivation is to enable the system to encode selectively various image areas and to produce perceptually pleasing coded images where faces are sharper. We refer to this approach—a mix of classical waveform coding and model—based coding-as model-assisted coding. We propose two totally automatic algorithms which, respectively, perform the detection of a head outline, and identify an ‘eyes-nose-mouth’ region, both from downsampled binary thresholded edge images. The algorithms operate accurately and robustly, even in cases of significant head rotation or partial occlusion by moving objects. We show how the information about face and facial feature location can be advantageously exploited by low bit-rate waveform-based video coders. In particular, we describe a method of object-selective quantizer control in a standard coding system based on motion-compensated discrete cosine transform—CCITT's recommendation H.261. The approach is based on two novel algorithms, namely buffer rate modulation and buffer size modulation. By forcing the rate control algorithm to transfer a fraction of the total available bit-rate from the coding of the non-facial to that of the facial area, the coder produces images with better-rendered facial features, i.e. coding artefacts in the facial area are less pronounced and eye contact is preserved. The improvement was found to be perceptually significant on video sequences coded at the ISDN rate of 64 kbps, with 48 kbps for the input (color) video signal in QCIF format.

We analyze the problem of optimal data partitioning of MPEG-2 coded video in an operational rate-distortion context. The optimal algorithm is characterized and shown to have high complexity and delay. A causally optimal algorithm based on Lagrangian optimization is proposed, that optimally solves the problem for intra (I) pictures, while it provides an optimal solution for predicted/interpolated (P/B) pictures when the additional constrains of causal operation and/or low-delay are imposed. A memoryless version of the algorithm, theoretically optimal for intra-pictures only, is shown to perform almost identically but with significantly less computational complexity. Finally, a fast, suboptimal algorithm using slice-based optimization is also proposed, and is shown to perform quite close (within 1 dB) to the causally optimal one.>

Repetitive image coding is encountered in applications such as video transcoding and image/video servers. This paper studies the effect of error accumulation in repetitive image coding. We describe the conditions under which zero-error-accumulation property holds. We study the impact of intermediate operations (like linear filtering, sealing, and shifting) on image quality in repetitive coding. Experiments with DCT, MC, and MPEG coding algorithms are reported.>

We present a novel and practical way to integrate techniques from computer vision to low bit rate coding systems for video teleconferencing applications. Our focus is to locate and track the faces of persons in typical head-and-shoulders video sequences, and to exploit the face location information in a "classical" video coding/decoding system. The motivation is to enable the system to selectively encode various image areas and to produce psychologically pleasing coded images where faces are sharper. We refer to this approach as model-assisted coding. We propose a totally automatic, low-complexity algorithm, which robustly performs face detection and tracking. A priori assumptions regarding sequence content are minimal and the algorithm operates accurately even in cases of occlusion by moving objects. Face location information is exploited by a low bit rate 3D subband-based video coder which uses a model-assisted dynamic bit allocation with object-selective quantization. By transferring a small fraction of the total available bit rate from the non-facial to the facial area, the coder produces images with better-rendered facial features. The improvement was found to be perceptually significant on video sequences coded at 96 kbps for an input luminance signal in CIF format. The technique is applicable to any video coding scheme that allows for fine-grain quantizer selection (e.g. MPEG, H.261), and can maintain full decoder compatibility.>

We describe and evaluate the performance of the algorithms used in Columbia University's "Xphone" multimedia communication system. The system assumes a "best-effort" operating system and network, and provides synchronized video/audio acquisition/playback (locally or across a network) with minimized and bounded end-to-end delay. Synchronization is achieved using an algorithm based on time-stamps and device state information. The effects of jitter (delay variation) are mitigated using silence detection; the end-to-end delay is kept bounded using a restart mechanism. Finally, for live video sources, we describe a source bit-rate adaptation algorithm that maximizes the video image quality to the available network bandwidth and video display window size.