Video communication is often afflicted by various forms of losses, such as packet loss over the Internet. This paper examines the question of whether the packet loss pattern, and in particular, the burst length, is important for accurately estimating the expected mean-squared error distortion resulting from packet loss of compressed video. We focus on the challenging case of low-bit-rate video where each P-frame typically fits within a single packet. Specifically, we: 1) verify that the loss pattern does have a significant effect on the resulting distortion; 2) explain why a loss pattern, for example a burst loss, generally produces a larger distortion than an equal number of isolated losses; and 3) propose a model that accurately estimates the expected distortion by explicitly accounting for the loss pattern, inter-frame error propagation, and the correlation between error frames. The accuracy of the proposed model is validated with H.264/AVC coded video and previous frame concealment, where for most sequences the total distortion is predicted to within plusmn0.3 dB for burst loss of length two packets, as compared to prior models which underestimate the distortion by about 1.5 dB. Furthermore, as the burst length increases, our prediction is within plusmn0.7 dB, while prior models degrade and underestimate the distortion by over 3 dB. The proposed model works well for video-telephony-type of sequences with low to medium motion. We also present a simple illustrative example, of how knowledge of the effect of burst loss can be used to adapt the schedule of video streaming to provide improved performance for a burst loss channel, without requiring an increase in bit rate.

The quality of service limitation of today's best-effort networks poses major challenge for low-latency video communication. To combat network losses for real-time and on-demand video communication, which exhibits stronger dependency across packets, a network-adaptive coding scheme is employed to dynamically manage the packet dependency using optimal reference picture selection. The selection of the reference is achieved within a rate-distortion optimization framework and is adapted to the varying network conditions. For network-adaptive streaming of prestored video, based on an accurate loss-distortion model, a prescient scheme that optimizes the dependency of a group of packets is proposed to achieve global optimality as well as improved rate-distortion performance. With the improved trade-off between compression efficiency and error resilience, the proposed system does not require retransmission of lost packets, which makes less than one-second low-latency communication possible.

Video communication is often afflicted by various forms of losses, such as packet loss over the Internet. The paper examines the question of whether the packet loss pattern, and in particular the burst length, is important for accurately estimating the expected mean-squared error distortion. Specifically, we (1) verify that the loss pattern does have a significant effect on the resulting distortion, (2) explain why a loss pattern, for example a burst loss, generally produces a larger distortion than an equal number of isolated losses, and (3) propose a model that accurately estimates the expected distortion by explicitly accounting for the loss pattern, inter-frame error propagation, and the correlation between error frames. The accuracy of the proposed model is validated with JVT/H.26L coded video and previous frame concealment, where for most sequences the total distortion is predicted to within /spl plusmn/0.3 dB for burst loss of length two packets, as compared to prior models which underestimate the distortion by about 1.5 dB. Furthermore, as the burst length increases, our prediction is within /spl plusmn/0.7 dB, while prior models degrade and underestimate the distortion by over 3 dB.

An improved, prescient, scheme to deliver preencoded video streams at very low latency is presented. We dynamically manage the prediction dependency for a group of packets using a rate-distortion (R-D) optimization framework that adapts to the channel, so that the expected end-to-end distortion is minimized under a given rate constraint. In our system, the R-D characteristics of each prediction mode are precomputed at compression time and stored in an R-D preamble along with the multiple compressed versions of the stream. The distortion in the preamble is estimated using an accurate loss-distortion model, and the optimal prediction modes are computed with an iterative descent algorithm. Complexity is reduced at streaming time by using the preamble to compute the optimal modes instead of actually compressing the video. Experiments demonstrate that the proposed prescient scheme provides significant performance gains over simple intra-insertion, and consistent bitrate savings compared to the greedy scheme.

Packet path diversity is one of the recent advances in network-adaptive video streaming. In this paper we first examine two specific techniques for video streaming that exploit diversity to achieve improved performance. The first technique uses a framework for rate-distortion optimized scheduling of the packet transmissions over the available network paths. The second technique exploits feedback and channel probing to determine which network path should be used for transmission and to adapt the source encoding of the video to mitigate error propagation effects. In the final section, we compare the performance of these two techniques by analyzing experimental results and discuss their respective advantages and drawbacks.

Low latency video applications need to incorporate a high level of error resiliency and of error concealment to provide acceptable quality. We investigate how to combine the benefits of multiple stream coding and multiple path transmission for this purpose. We present a transmission strategy for multiple description coded video which maintains the inherent error resiliency of the encoding in the case of transmission over bursty channels. Through the analysis of feedback and through rate-distortion optimized channel probing, this strategy determines which channels should be used to take advantage of path diversity. At the encoder, a restart scheme stops error propagation by selecting references in error-free streams. Experiments demonstrate significant gains over other schemes such as video redundancy coding.

The quality of service limitation of today's Internet is a major challenge for real-time voice communications. Excessive delay, packet loss, and high delay jitter all impair the communication quality. A new receiver-based playout scheduling scheme is proposed to improve the tradeoff between buffering delay and late loss for real-time voice communication over IP networks. In this scheme the network delay is estimated from past statistics and the playout time of the voice packets is adaptively adjusted. In contrast to previous work, the adjustment is not only performed between talkspurts, but also within talkspurts in a highly dynamic way. Proper reconstruction of continuous playout speech is achieved by scaling individual voice packets using a time-scale modification technique based on the Waveform Similarity Overlap-Add (WSOLA) algorithm. Results of subjective listening tests show that this operation does not impair audio quality, since the adaptation process requires infrequent scaling of the voice packets and low playout jitter is perceptually tolerable. The same time-scale modification technique is also used to conceal packet loss at very low delay, i.e., one packet time. Simulation results based on Internet measurements show that the tradeoff between buffering delay and late loss can be improved significantly. The overall audio quality is investigated based on subjective listening tests, showing typical gains of 1 on a 5-point scale of the Mean Opinion Score.

We review recent advances in channel-adaptive video streaming. Adaptive media playout at the client can be used to reduce receiver buffering and therefore average latency, and provide a limited rate scalability. Rate-distortion optimized packet scheduling determines the best packet to send given the distortion reduction associated with sending that packet, interpacket dependencies, and the success of past transmissions. Channel-adaptive packet dependency control can greatly improve the error-robustness of streaming video and reduce or eliminate the need for packet retransmissions. Finally we consider three architectures for wireless video streaming and discuss the utility of the discussed techniques for each architecture.

Today's Internet video streaming systems employ buffering and multiple retransmissions to guarantee the correct reception of each packet, which leads to high latency in media delivery. In this work, a novel scheme of dynamic management of the dependency across packets is proposed using optimal reference picture selection at the frame level that adapts to the channel. The optimal selection of the reference picture is achieved with a rate-distortion framework, which minimizes the expected end-to-end distortion given a rate constraint. The expected distortion is calculated based on an accurate binary tree modeling with the effects of channel loss and error concealment taken into account. Experiments demonstrate that the proposed :scheme provides significant performance gains over a simple INTRA-insertion scheme. The increased error-resilience eliminates the need of retransmission, which makes it possible to reduce the latency from 10-15 seconds to a few hundred milliseconds, with similar video quality maintained.

Today's Internet video streaming systems employ buffering and retransmission to guarantee the correct reception of each packet. This leads to high latency in media delivery. In this paper, we present an efficient low-latency Internet video streaming system that does not require retransmission of lost packets. We pre-store multiple representations of certain frames on the server such that a representation using a more reliable reference frame will be selected and sent during transmission. The potential mismatch error during bitstream assembly at transmission is avoided by using a novel layered coding structure with coding restrictions. The optimal picture type is determined within a rate-distortion framework during pre-encoding and transmission, adapting to the channel, so that the expected end-to-end distortion is minimized under the given rate constraint. The expected distortion is calculated based on an accurate binary tree modeling with the effects of channel loss and error concealment taken into account. Experiments demonstrate that the proposed scheme provides significant performance gains over a simple INTRA-insertion scheme. The increased error-resilience eliminates the need for retransmission, which allows to reduce the latency of streaming from 10-15 seconds to a few hundred milliseconds, with good video quality maintained.

Abstract Despite the well‐known challenges of variations in throughput, delay, and packet loss over the Internet, video streaming has experienced phenomenal growth, owing to the extensive research in video coding and transmission. In this paper, we review several recent advances for channel‐adaptive video streaming that, we believe, will benefit the design of video streaming systems in the future. Employed in different components of the system, these techniques have the common objective of providing efficient, robust, scalable, and low‐latency streaming video. Firstly, by allowing the client to control the rate at which it consumes data, adaptive media playout can be used to reduce receiver buffering and therefore average latency, and provide limited rate scalability. Secondly, rate‐distortion optimized packet scheduling, a transport technique, provides a flexible framework to determine the best packet to send, given the channel behaviors, the packets' deadlines, their transmission histories, the distortion reduction associated with sending each packet, and the interpacket dependencies. Thirdly, at the source encoder, channel‐adaptive packet‐dependency control can greatly improve the error resilience of streaming video and reduce latency. Finally, we address the specific additional challenges for wireless video streaming. We consider three architectures for wireless video and discuss the utility of the reviewed techniques for each architecture. Copyright © 2002 John Wiley & Sons, Ltd.

A new receiver-based playout scheduling scheme is proposed, which estimates the network delay from past statistics and adaptively adjusts the playout time of the voice packets. In contrast to previous work, the adjustment is not only performed in between talkspurts, but also within the talkspurts in a highly dynamic way. Proper reconstruction of continuous output speech is achieved by scaling individual voice packets using a time-scale modification technique which modifies the rate of playout while preserving voice pitch. Subjective listening tests show that this operation does not impair the audio quality. Simulation results based on Internet measurement indicate that buffering delay and loss rate can be significantly reduced by adaptive scheduling.

The quality of real-time voice communication over best-effort networks is mainly determined by the delay and loss characteristics observed along the network path. Excessive playout buffering at the receiver is prohibitive and significantly delayed packets have to be discarded and considered as late loss. We propose to improve the tradeoff among delay, late loss rate, and speech quality using multi-stream transmission of real-time voice over the Internet, where multiple redundant descriptions of the voice stream are sent over independent network paths. Scheduling the playout of the received voice packets is based on a novel multi-stream adaptive playout scheduling technique that uses a Lagrangian cost function to trade delay versus loss. Experiments over the Internet suggest largely uncorrelated packet erasure and delay jitter characteristics for different network paths which leads to a noticeable path diversity gain. We observe significant reductions in mean end-to-end latency and loss rates as well as improved speech quality when compared to FEC protected single-path transmission at the same data rate. In addition to our Internet measurements, we analyze the performance of the proposed multi-path voice communication scheme using the ns network simulator for different network topologies, including shared network links.

We propose multi-stream transmission of real-time voice over best-effort packet networks such as today's Internet, where multiple redundant descriptions of the voice stream are sent over independent network paths. At the receiver, multi-stream adaptive playout scheduling is employed to improve the tradeoff among delay, late loss rate, and speech quality. Experiments over the Internet suggest largely uncorrelated statistical characteristics, such as erasure probability and delay jitter, for different network paths, which leads to a noticeable path diversity gain. We have obtained significant reductions in mean end-to-end latency and loss rates compared to FEC protected single-path transmission at the same data rate. The speech quality perceived by the receiver is evaluated using the recently standardized objective quality measure PESQ. In our experiments, we observe gains of more than 0.4 PESQ score for voice transmission with packet path diversity.