Characterizing the video quality seen by an end-user is a critical component of any video transmission system. In packet-based communication systems, such as wireless channels or the Internet, packet delivery is not guaranteed. Therefore, from the point-of-view of the transmitter, the distortion at the receiver is a random variable. Traditional approaches have primarily focused on minimizing the expected value of the end-to-end distortion. This paper explores the benefits of accounting for not only the mean, but also the variance of the end-to-end distortion when allocating limited source and channel resources. By accounting for the variance of the distortion, the proposed approach increases the reliability of the system by making it more likely that what the end-user sees, closely resembles the mean end-to-end distortion calculated at the transmitter. Experimental results demonstrate that variance-aware resource allocation can help limit error propagation and is more robust to channel-mismatch than approaches whose goal is to strictly minimize the expected distortion.

The problem of application-layer error control for real-time video transmission over packet lossy networks is commonly addressed via joint source-channel coding (JSCC), where source coding and forward error correction (FEC) are jointly designed to compensate for packet losses. In this paper, we consider hybrid application-layer error correction consisting of FEC and retransmissions. The study is carried out in an integrated joint source-channel coding (IJSCC) framework, where error resilient source coding, channel coding, and error concealment are jointly considered in order to achieve the best video delivery quality. We first show the advantage of the proposed IJSCC framework as compared to a sequential JSCC approach, where error resilient source coding and channel coding are not fully integrated. In the USCC framework, we also study the performance of different error control scenarios, such as pure FEC, pure retransmission, and their combination. Pure FEC and application layer retransmissions are shown to each achieve optimal results depending on the packet loss rates and the round-trip time. A hybrid of FEC and retransmissions is shown to outperform each component individually due to its greater flexibility.

Multimedia applications involving the transmission of video over communication networks are rapidly increasing in popularity. Such applications can greatly benefit from adapting video coding parameters to network conditions as well as adapting network parameters to better support the application requirements. These two dimensions can both be viewed as allocating source and network resources to improve video quality. We highlight recent advances in optimal resource allocation for real-time video communications over unreliable and resource constrained communication channels. More specifically, we focus on point-to-point coding and delivery schemes in which the sequences are encoded on the fly. We present a high-level framework for resource-distortion optimization. The framework can be used for jointly considering factors across network layers, including source coding, channel resource allocation, and error concealment. For example, resources can take the form of transmission energy in a wireless channel, and transmission cost in a DiffServ-based Internet channel. This framework can be used to optimally trade off resource consumption with end-to-end video quality in packet-based video transmission. After giving an overview of this framework, we review recent work in two areas-energy efficient wireless video transmission and resource allocation for Internet-based applications.

Object-based video coding is a relatively new technique to meet the fast growing demand for interactive multimedia applications. Compared with conventional frame-based video coding, it consists of two types of source data: shape information and texture information. Recently, joint source-channel coding for multimedia communications has gained increased popularity. However, very limited work has been conducted to address the problem of joint source-channel coding for object-based video. In this paper, we propose a cost-distortion optimal unequal error protection (UEP) scheme for object-based video communications. Our goal is to achieve the best video quality (minimum total expected distortion) with constraints on transmission cost and delay in a lossy network environment. The problem is solved using Lagrangian relaxation and dynamic programming. The performance of the proposed scheme is tested using simulations of a narrow-band block-fading wireless channel with additive white Gaussian noise and a simplified differentiated services Internet channel. Experimental results indicate that the proposed UEP scheme can significantly outperform equal error protection methods.

Differentiated Services (DiffServ) is one of the leading architectures for providing quality of service in the Internet. We propose a scheme for real-time video transmission over a DiffServ network that jointly considers video source coding, packet classification, and error concealment within a framework of cost-distortion optimization. The selections of encoding parameters and packet classification are both used to manage end-to-end delay variations and packet losses within the network. We present two dual formulations of the proposed scheme: the minimum distortion problem, in which the objective is to minimize the end-to-end distortion subject to cost and delay constraints, and the minimum cost problem, which minimizes the total cost subject to end-to-end distortion and delay constraints. A solution to these problems using Lagrangian relaxation and dynamic programming is given. Simulation results demonstrate the advantage of jointly adapting the source coding and packet classification in DiffServ networks.

The problem of encoding and transmitting a video sequence over an IP-based wireless network, consisting of both wired and wireless links, is addressed. To combat the different types of packet loss in the heterogeneous network, the use of a product code forward error correction (FEC) scheme capable of providing unequal error protection is considered. At the transport layer, Reed-Solomon (RS) coding is used to provide inter-packet protection. In addition, rate-compatible punctured convolutional (RCPC) coding is used at the link layer to provide unequal intra-packet protection. Optimal bit allocation is performed in a rate-distortion optimized joint source-channel coding and power allocation framework to achieve the best video quality. Simulation results illustrate the advantage of the proposed product code FEC scheme over previously studied approaches.

Fine granular scalability is a coding tool, recently introduced in the emerging MPEG-4 standard, which enables the creation of very flexible scalable video bitstreams. This paper investigates the transmission of fine granular scalable (FGS) video over wireless links, using power management for unequal error protection of the bitstream. In wireless systems, energy may be a limited resource, and a wise use of it is important for system efficiency. An algorithm is proposed which is able to optimally distribute the total available power for the transmission of the enhancement layer, given a distortion or energy constraint. Experimental results demonstrate the performance advantage of the proposed algorithm over fixed power schemes and heuristic approaches.

In this paper, hybrid error control for real-time video transmission is studied. The study is carried out using a proposed integrated joint source-channel coding framework, which jointly considers error resilient source coding, channel coding, and error concealment, in order to achieve the best video quality and focuses on the performance comparison of several error correction scenarios, such as forward error correction (FEC), retransmission, and the combination of both. Simulation results show that either FEC or retransmission can be optimal depending on the packet loss rates and network round trip time. The proposed hybrid FEC/retransmission scheme outperforms both.

A major limitation faced by a mobile user is their dependence on a limited battery supply. For wireless video communications, joint source coding and transmission power management (JSCPM) has recently been considered as a means of efficiently allocating transmission energy. In order to reduce complexity, the design of many of these adaptive resource allocation algorithms utilizes simplified channel models that do not account for the burstiness of the channel. We analyze the effects of such channel model simplifications on the end-to-end distortion. We present a channel model that is based on information theoretic considerations, which captures the bursty nature of wireless channels and accounts for packet lengths when calculating the probability of loss. Given the source coding and transmission parameters derived using a simplified channel model, our goal is to analyze how the end-to-end distortion is affected when a more realistic complex channel model is used to simulate losses. Experimental results suggest that the performance gain predictions for JSCPM using a simpler channel model are also valid when more sophisticated channel simulations are used, provided that a number of additional steps are taken after the optimization to account for the complex characteristics of wireless channels.

In this paper, we consider dynamic resource allocation for object-based wireless video communications. In object-based video coding, a video frame is comprised of objects that are described by their shape as well as their texture. By jointly considering source coding, error concealment and transmission power management at the physical layer, the proposed framework minimize the expect distortion at the receiver for given energy and delay constraints. In order to provide unequal error protection for the shape and texture information, a new video packetization scheme is proposed. Experimental results indicate that the proposed unequal error protection schemes significantly outperform equal error protection methods.

In this paper, we propose an optimal unequal error protection scheme for object-based video communications over differentiated services networks. Our goal is to achieve the best video quality (minimum total expected distortion) with constraints on transmission cost and delay. An end-to-end distortion estimation approach for object-based video is proposed, which can be used for different packetization schemes. The problem is solved using Lagrangian relaxation and dynamic programming. Experimental results indicate that the proposed unequal error protection schemes can significantly outperform equal error protection methods.

The problem of application-layer error control for real-time video transmission over packet lossy networks is commonly addressed by joint source-channel coding (JSCC). The traditional JSCC approaches solve this problem in a sequential manner, where source coding and channel coding are not fully integrated. In this paper, we present an integrated joint source-channel coding (IJSCC) framework, where error resilient source coding, channel coding and error concealment are jointly considered in an integrated manner. We show through both analysis and simulations the advantages of the proposed IJSCC approach, in comparison to a sequential JSCC approach.

The problem of encoding and transmitting a video sequence over a wireless channel is considered. Our objective is to minimize the end-to-end distortion while using a limited amount of transmission energy and delay. In our approach, we jointly adapt the source-coding parameters and transmission power per packet. We introduce the concept of "variance-aware distortion estimation" (VADE), and present a framework for controlling both the expected value and the variance of the end-to-end distortion. This framework is based on knowledge of how the video is compressed, the probability of packet loss, and the concealment strategy. To the best of our knowledge, this paper is the first to address the trade-off between the mean and variance of the end-to-end distortion. Experimental results demonstrate the potential of the proposed approach.

This paper presents a novel framework for streaming video over a Differentiated Services (DiffServ) network that jointly considers video source coding, packet classification and error concealment within the scope of cost-distortion optimization. Our formulation incorporates the random network delay for each packet into the calculation of the probability of packet loss and manages the end-to-end packet delay by selecting the encoding parameters and packet priority. We formulate two approaches to evaluate the performance of the proposed framework: a minimum distortion approach and a minimum cost approach, in which the encoding mode and priority class for each packet are optimally selected so as to minimize the total distortion subject to cost constraints, or to minimize the total cost subject to end-to-end distortion constraints. Simulation results demonstrate the advantage of jointly adapting the source coding and packet classification.

Rapid growth in wireless networks is fueling demand for video services from mobile users. While the problem of transmitting video over unreliable channels has received some attention, the wireless network environment poses challenges such as transmission power management that have received little attention previously in connection with video. Transmission power management affects battery life in mobile devices, interference to other users, and network capacity. We consider energy efficient transmission of a video sequence under delay and quality constraints. The selection of source coding parameters is considered jointly with transmitter power and rate adaptation, and packet transmission scheduling. The goal is to transmit a video frame using the minimal required transmission energy under delay and quality constraints. Experimental results are presented that illustrate the advantages of the proposed approach.

In the future, digital set-top boxes may serve as the primary access point for wireless home networks, enabling mobile users to use videoconferencing as well as streaming applications on hand-held devices. In this scenario, an important issue that must be addressed is the limited energy supply of a mobile device. This is of course a relevant issue for any wireless device. We focus on methods for efficiently utilizing transmission energy in wireless video communications. We present a general framework for the problem of minimizing the transmission energy required to provide an acceptable level of video quality. We discuss two special cases in which communication resources are adjusted simultaneously with the source coding parameters in order to provide (i) packet loss adaptation and (ii) transmission rate adaptation.

We consider the problem of compressing a video sequence for transmission over a wireless channel. In our approach we jointly consider error resilience and concealment techniques, at the source coding level, and transmission power management at the physical layer. We formulate a minimum-maximum distortion problem, where our goal is to either (i) minimize the total transmission energy for a given maximum expected distortion, or (ii) minimize the maximum expected distortion at the receiver for a given maximum transmission energy. Experimental results show that simultaneously adjusting the source coding and transmission power is more energy efficient than considering these factors separately.

We consider a situation where a video sequence is to be compressed and transmitted over a wireless channel. Our goal is to limit the amount of distortion in the received video sequence, while minimizing transmission energy. To accomplish this goal, we consider error resilience and concealment techniques at the source coding level, and transmission power management at the physical layer. We jointly consider these approaches in a novel framework. In this setting, we formulate and solve an optimization problem that corresponds to minimizing the energy required to transmit video under distortion and delay constraints. Experimental results show that simultaneously adjusting the source coding and transmission power is more energy efficient than considering these factors separately.

A key constraint in mobile communications is the reliance on a battery with a limited energy supply. Efficiently utilizing the available energy is therefore an important design consideration. We consider a situation where a video sequence is to be compressed and transmitted over a wireless channel. The goal is to limit the amount of distortion in the received video sequence while using the minimum required transmission energy. To accomplish this goal, we consider error resilience and concealment techniques, at the source coding level, as well as the dynamic allocation of physical layer communication resources. We consider these approaches jointly in a novel framework. We formulate an optimization problem that corresponds to minimizing the energy required to transmit a video frame with an acceptable level of distortion. We present methods for solving this problem and other extensions.