In this paper, we exploit the stylistic characteristics of high-quality entertainment movie sequences in terms of their textured background for coding purposes. More specifically, we propose a content-based coding method by texture replacement. At the encoder, texture is removed from selected regions of the original frames. The resulting frames with the texture removed and the parameters of the removed texture are then encoded. At the decoder, the boundaries of the regions without texture are identified and new texture, which is synthesized using the decoded texture parameters, is mapped onto these regions. Our experimental results confirm the main advantages of the proposed texture replacement method: significant bit rate reduction of the compressed movie sequences with the texture removed, and higher visual quality of the textured background regions in the decoded movie sequences with synthesized texture than that of the regions in the sequences simply encoded and decoded. Even more, our method can be applied as an overlay onto any standards-compliant coding system.

In this paper, we propose a method for direct mode enhancement in B pictures in the framework of H.264 (MPEG-4/Part 10), which employs reduction of outliers, clustering of the Lagrangian values and specification of smaller values of the Lagrangian multiplier in the rate-distortion optimization for mode selection. Experimental results using high quality video sequences show that bit rate reduction and no subjective visual loss in the decoded pictures are obtained using our method. Moreover, we verify that our method achieves further bit rate reduction without introducing visible distortion in the decoded sequences when other solutions would introduce inadmissible artifacts. Last but not least, the proposed method is a simple and useful add-on, which is applicable in any video coding system that employs rate-distortion optimization for coding mode selection.

We propose a method for texture replacement in video sequences. Our method, which is applied at the encoder side, consists of removal of texture from selected regions of the original frames, synthesis of new texture, and mapping of the new texture back onto the segmented regions. The texture removal stage employs highly effective color-based angular maps. The texture analysis and texture synthesis stages make use of steerable pyramids. The latter stage also employs constraints that are derived using a vocabulary and grammar for color pattern similarity evaluation that have been introduced previously. Because they have different characteristics than those of the original textures, the synthesized textures can be coded more effectively. Consequently and most importantly, significantly reduced bit rates of the compressed video sequences with the texture replaced are obtained as compared to those of the original sequences. Moreover, because the synthesized textures have similar perceptual characteristics to those of the original textures, the video sequences with the texture replaced are also visually similar to the original sequences. Even more, because it is performed at the encoder and it does not have any impact on the decoder, our texture replacement method is cost effective. We illustrate its performance and computational efficiency using movie sequences.

In this paper, we propose a method for texture replacement in video sequences at the encoder. Our method consists of removal of texture from selected regions of the original frames, analysis of the removed texture, synthesis of new texture, and mapping of the new texture back onto the segmented regions. Our proposed method yields significantly reduced bit rates of the compressed video sequences with the texture replaced as compared to those of the original sequences. Moreover, the video sequences with texture replacement are visually similar to the original sequences. Even more, our decoder-independent method is cost effective.

We propose a content-based method for coding entertainment movie sequences using texture replacement at the encoder and texture synthesis and mapping at the decoder. Our method reduces the bit rate of the compressed movie sequences significantly and yields higher visual quality of the textured background regions in the decoded movie sequences with mapped texture than that of the regions in the sequences simply encoded and decoded. Moreover, our method is efficient in terms of speed and may be applied as an overlay onto any standards-compliant coding system.

First, a brief overview of the MPEG-4 standard is presented to highlight the fact that in order to achieve object based functionality, it requires arbitrary shaped visual objects. Next, a number of applications and their requirements are reviewed to determine the need for more than one class of solutions for visual shape representation. Then, a brief review of the solution for shape representation in MPEG-4 version 1 is presented, and, a solution that we have proposed for MPEG-4 version 2 is outlined. Details of our implicit shape coding which uses chromakey in MPEG4 systems scene description (BIFS) are then presented. Implementation of our shape representation in a real-time MPEC-4 player is then discussed. Finally we address the context in which our shape representation can address unique functionalities.

The challenge of multimedia processing is to provide services that seamlessly integrate text, sound, image, and video information and to do it in a way that preserves the ease of use and interactivity of conventional plain old telephone service (POTS) telephony. To achieve this goal, there are a number of technological problems that must be considered, including: compression and coding of multimedia signals, including algorithmic issues, standards issues, and transmission issues; synthesis and recognition of multimedia signals, including speech, images, handwriting, and text; organization, storage, and retrieval of multimedia signals, including the appropriate method and speed of delivery, resolution, and quality of service; access methods to the multimedia signal, including spoken natural language interfaces, agent interfaces, and media conversion tools; searching by text, speech, and image queries; browsing by accessing the text, by voice, or by indexed images. In each of these areas, a great deal of progress has been made in the past few years, driven in part by the relentless growth in multimedia personal computers and in part by the promise of broad-band access from the home and from wireless connections. Standards have also played a key role in driving new multimedia services, both on the POTS network and on the Internet. It is the purpose of this paper to review the status of the technology in each of the areas listed above and to illustrate current capabilities by describing several multimedia applications that have been implemented at AT&T Labs over the past several years.

Discusses coding standards for still images and motion video. We first briefly discuss standards already in use, including: Group 3 and Group 4 for bilevel fax images; JPEG for still color images; and H.261, H.263, MPEG-1, and MPEG-2 for motion video. We then cover newly emerging standards such as JBIG1 and JBIG2 for bilevel fax images, JPEG-2000 for still color images, and H.263+ and MPEG-4 for motion video. Finally, we describe some directions beyond the standards such as hybrid coding of graphics/photo images, MPEG-7 for multimedia metadata, and possible new technologies.

The challenge of multimedia processing is to seamlessly integrate text, sound, image, and video information into a single communications channel, and to do it in a way that provides high quality communications while preserving the ease-of-use and interactivity of conventional telephony. There are a number of technology drivers that are pushing the technology forward, as well as a number of technological problems that must be overcome before multimedia becomes as ubiquitous as voiceband telephony. A key issue with any practical multimedia system has to do with standards that insure connectivity between customers and a range of service providers. Multimedia processing is an area of communications that is rapidly evolving. However, a number of interesting and important multimedia communications applications have evolved over the past several years, and some of these applications are described.

This paper presents a method to color balance images. This method is useful for correcting images in video conferencing, where multiple video sources are displayed and each image has a different "look". The advantages of this approach are that it does not require a color reference object in each image, i.e. no a priori knowledge of the scene is required, that it can be used on the entire image, and that it is simple to implement. Preliminary results show that our approach can successfully correct images to appear more uniform and visually pleasing.

We propose a scheme for coding subregions in video scenes to provide content-based scalable video. For each region, a special color is used to represent the nonobject area, and the resulting frames are coded using conventional video coding algorithms. At the decoder, the region shape is recovered based on chroma keying, and hence, content-based manipulations are made possible. A number of techniques that eliminate boundary artifacts common to region-based coding are presented. In this scheme, no explicit shape coding is needed, and advantages of existing coding algorithms are retained. This scheme was submitted to ISO MPEG-4 and performed very well in the subjective tests.

We propose a scheme for coding sub-regions in video scenes to provide content-based scalable video. For each region, a special color is used to represent the non-object area, and the resulting frames are coded using conventional video coding algorithms. At the decoder, the region shape is recovered based on chroma keying, and hence content-based manipulations are made possible. A number of techniques that eliminate boundary artifacts common to region-based coding are presented. In this scheme, no explicit shape coding is needed, and the advantages of existing coding algorithms are retained. This scheme was submitted to ISO MPEG-4, and performed very well in the subjective tests.

We utilize speech information to improve the quality of audio-visual communications such as video telephony and videoconferencing. We show that the marriage of speech analysis and image processing can solve problems related to lip synchronization. We present a technique called speech-assisted frame-rate conversion, and apply it to coding of talking head video. Demonstration sequences are presented. Extensions and other applications are outlined.

A motion compensated video coding technique is discussed where square macro-blocks in the coding process are replaced by arbitrary shaped regions. Different components of this technique are presented in detail, such as: segmentation, region shape coding, motion estimation, and mode decisions. Simulation results show good quality images with sharper details achieved at bitrates as low as 9.6 Kb/s.

This paper describes a method for coding arbitrarily shaped image segments. The method uses an iterative technique based on the theory of successive projection onto convex sets to determine the best transform coefficients. It uses block transforms with frequency domain region-zeroing and space domain region-enforcing operations for effective coding of image segments of arbitrary shape. A major strength of this method is that it can be implemented in real-time using existing codec hardware at an insignificant additional cost.>

Scalable video coding is important in a number of applications where video needs to be decoded and displayed at a variety of spatial and temporal resolution scales or when resilience to errors is necessary. In this paper, we investigate a temporal-domain approach for resolution scalable video coding. This approach was originally proposed to and since then has been included in the Moving Picture Experts Group (MPEG-2) standard. Temporal scalability, although not limited to, is particularly useful for applications such as, HDTV as well as for high quality telecommunication applications that may require high frame rate (e.g., 60 Hz) progressive video. Furthermore, it allows flexibility in use of progressive or interlaced video formats for coding even though the source video may be progressive with high frame rate. Our technique builds on the motion-compensated DCT framework of nonscalable (single layer) MPEG-2 video coding and adds motion compensated inter-layer coding. In our simulations we focus on the 2- layer temporally scalable coding and compare the performance of using progressive video format for both layers to that when interlaced video format is used for both layers; all interim video formats used are derived from original high temporal resolution progressive video. In each case we compare the performance of two inter-layer prediction configurations and evaluate their prediction efficiency.>

Discusses the problem of multiplexing several variable rate encoded streams into a single stream. Moreover, in order to facilitate editing of the resulting stream it is required that data generated during the same time interval is multiplexed together. Particular emphasis is placed on controlling encoder rates and combining data in such a way as to avoid overflow and underflow of buffers at encoder and decoder. Applications include satellite or cable transmission of a fixed number of different video channels, multimedia presentations with multiple video streams, and video on demand.>

Constraints on the encoded bit rate of a video signal that are imposed by a channel and encoder and decoder buffers are considered. Conditions that ensure that the video encoder and decoder buffers do not overflow or underflow when the channel can transmit a variable bit rate are presented. Using these conditions and a commonly proposed network-user contract, the effect of a (BISDN) network policing function on the allowable variability in the encoded video bit rate is examined. It is shown how these ideas might be implemented in a system that controls both the encoded and transmitted bit rates. The performance of video that has been encoded using the derived constraints for the leaky bucket channel is presented.>

The authors suggest a parallel processing structure using the proposed international standard for visual telephony (CCITT P*64 kbs standard) as processing elements, to compress digital high definition television (HDTV) pictures. The basic idea is to partition an HDTV picture, in space or in frequency, into smaller sub-pictures and then compress each sub-picture using a CCITT P*64 kbs coder. This seems to be a cost-effective solution to the HDTV hardware. Since each sub-picture is processed by an independent coder, without coordination these coded sub-pictures may have unequal picture quality. To maintain a uniform quality HDTV picture, the following two issues are studied: sub-channel control strategy (bits allocated to each sub-picture); and quantization and buffer control strategy for individual sub-picture coders. Algorithms to resolve these problems and their computer simulations are presented.>

High Definition Television (HDTV) promises to offer wide-screen, much better quality pictures as compared to the today’s television. However, without compression a digital HDTV channel may cost up to one Gbits/sec transmission bandwidth. We suggest a parallel processing structure using the proposed international standard for visual telephony (CCITT Px64 kbs standard) as processing elements, to compress the digital HDTV pictures. The basic idea is to partition an HDTV picture into smaller sub-pictures and then compress each sub-picture using a CCITT Px64kbs coder, which is cost-effective, by today’s technology, only on small size pictures. Since each sub-picture is processed by an independent coder, without coordination these coded sub-pictures may have unequal picture quality. To maintain a uniform quality HDTV picture, the following two issues are studied: (l) sub-channel control strategy (bits allocated to each sub-picture), and (2) quantization and buffer control strategy for individual sub-picture coder. Algorithms to resolve the above problems and their computer simulations are presented.

Three methods are investigated for improving the image quality obtainable by transform coding at relatively low bit rates. Coefficients are transmitted in the so-called zig-zag scanning order, which approximates an ordering according to increasing spatial frequency. For most blocks, coefficients higher than a certain order all have zero quantized value. These are not sent. Instead an end-of-block word is transmitted to indicate to the receiver that the remaining quantized coefficients of that block were zero. Interpolative transform coding requires side information, whereas predictive transform coding does not. Pyramid transform coding performs the best, but at the expense of considerable additional signal processing.>

Interpolative vector quantization has been devised to alleviate the visible block structure of coded images plus the sensitive codebook problems produced by a simple vector quantizer. In addition, the problem of selecting color components for color picture vector quantization is discussed. Computer simulations demonstrate the success of this coding technique for color image compression at approximately 0.3 b/pel. Some background information on vector quantization is provided.>

In a previous paper we proposed transmitting high definition television (HDTV) by means of semicompatible HDTV (SC-HDTV) signals which utilized differences between scan lines. The line-differential SC-HDTV waveform had the virtue of being easily convertible to an NTSC compatible color picture. The line-differential SC-HDTV signal required three present-day CATV channels for local distribution. With the advent of inexpensive electronics and memory, the possibility arises of incorporating field memories into HDTV receivers. Thus, in this paper we propose a field-differential SC-HDTV signal, which, like its predecessor, is easily converted to NTSC. However, the field-differential SC-HDTV waveform may be transmitted using only two present-day CATV channels. In order to produce an HDTV picture from the field-differential SCHDTV signal, the receiver requires comb filters, 2:1 time compressors and a field memory of about 245 kilopixels. These are implementable using today's technology.

We describe an experimental system for demonstrating time-compression multiplexing (TCM) of two NTSC color television signals in a satellite channel of 36 MHz. The system employs digital processing to derive line or field differentials from each picture. The television signals are then converted into interleaved lines (or fields) of unimpaired baseband video and companded line (or field) differentials. These signal components are finally time compressed and multiplexed into a combined signal for single-carrier FM transmission. With 4.5-m earth stations, the field-differential technique offers extremely good transmission quality suitable for TV distribution to cable head-ends (weighted signal-to-noise ratio, WSNR ≈ 51.5 dB), while the line-differential method provides a slightly lower WSNR ( ≈ 49 dB). We recommend the field-differential approach because of its superior overall picture quality. For larger receive stations (7m), higher picture quality (WSNR ≈ 56 dB) could be obtained. If 10-m earth stations are employed, the received video performance is practically indistinguishable from the corresponding one in the one-television-per-transponder case, and we infer that three pictures can indeed be sent with a graceful degradation as previously suggested. By choosing the parameters properly, the current TCM system can be optimized for a wide variety of applications with higher channel capacity.

High definition television (HDTV) pictures are, by all accounts, dramatically superior to today's NTSC color television, both in resolution and in realism. Picture quality approaching that of 35 mm color film is obtainable by most systems under investigation in recent years. One of the basic premises of a recent SMPTE study was that "the HDTV signal should be capable of a standards conversion ... to the NTSC ... domestic service." Otherwise, there would be "so many political, social, and economic Impediments that [HDTV] might never succeed ...." However, the same study wound up by concluding that such compatibility was "not feasible by any means known ... or envisaged." Here we suggest a possible solution to the compatibility dilemma. We propose an HDTV signal with a camera line-scan rate exactly twice that of NTSC. From this we construct an intermediate waveform called the semicompatible-high-definition television signal (SCHDTV), which has the virtue of being easily convertible to an NTSC picture. The SC-HDTV signal requires three present-day CATV channels for local distribution and is suitable for transmission via direct broadcast satellite (DBS). Moreover, HDTV could also be sent on most present-day satellite facilities if the SC-HDTV signal were split into two parts and two transponders were used.

We describe a time-frequency multiplexing (TFM) method for transmitting two television signals over one satellite transponder, which gives significant improvement over existing methods. The complexity of the hardware implementation is similar to existing time-division multiplexing (TDM) systems, but the picture quality, when received on a 10 m earth station, approaches that which is currently being requested by the major television networks. Unlike existing frequency-division multiplexing (FDM) systems, TFM needs no backoff of the satellite transponder. The general philosophy of TFM is described, including the benefits of using a difference signal to reduce the overall required bandwidth. Also discussed is companding of the difference signal as a means of improving the overall signal-to-noise ratio. Several alternative methods of implementation are considered, along with those which have been explored in some detail. We then describe an experimental TFM system which was implemented using digital field memories and tested with a 10 m earth station operating through the COMSTAR D2 satellite. Received picture quality was excellent in all cases. Weighted SNR was 53-56 dB, and in practically all instances subjects could not tell any difference between single-picture FM transmission and TFM dual-picture transmission.

This paper considers frequency hopped-multilevel frequency shift keyed (FH-MFSK) spread spectrum communication with possible application to mobile telephone service. Comparisons between randomly chosen address vectors, chirp vectors, and Einarsson's finite field vectors revealed that the latter two were equivalent (as expected) and somewhat better than random coding; however, the difference diminishes as bandwidth increases. Simple FH-MFSK yielded efficiencies around 0.36 bits/s/Hz for bandwidths in the range 3-20 MHz. A checking procedure is also described which, by a closer examination of the structure of the received data, is able to reduce errors markedly. It enables efficiencies around 0.55 bits/s/Hz for bandwidths in the range 3-20 MHz, which is close to that achievable via convolutional coding using dual-kcodes.

Techniques are investigated for conversion between different sampling rates for NTSC color television signals. It is concluded that sampling rate conversion between 3 times color subcarrier frequency and 4 times color subcarrier frequency using all digital processing is quite feasible, and that hardware requirements are not unreasonable. Also, accumulation of error due to cascaded sampling rate conversion can be avoided by use of appropriate finite-impulse-response digital filters.

Interframe coder simulation results are described for standard-broadcast-rate monochrome television in applications where the television camera is largely stationary, such as TV-conferencing or video telephone. An algorithm for coding at 1.5 Mbits/s, whose parameters were optimized via trial and error, produces pictures with little resolution loss for low-to-moderate movement, but somewhat visible moving-area resolution loss for higher movements.

In a conditional replenishment coder for videotelephony only those picture elements (pels) which have changed significantly since the previous frame are transmitted. The amplitudes as well as the locations of these pels within the frame must be sent. In [2] locations are sent by addressing clusters of changed pels with respect to the beginning of the line. Increased addressing efficiency results if clusters of changes are positioned not with respect to the beginning of the line, but with respect to another cluster in the previous line or previous frame which has already been transmitted. Computer simulations using digitally stored videotelephone signals were carried out to evaluate such differential schemes. Results showed that differential addressing of this type and the use of variable word-length codes can yield about a 50 percent saving in bits required to address beginnings of clusters. The overall bit rate saving is much smaller, however, ranging from 10-15 percent depending on the efficiency with which pel amplitudes are coded.