We propose a novel solution to the problem of robust, low-latency video transmission over lossy channels. Predictive video codecs, such as MPEG and H.26x, are very susceptible to prediction mismatch between encoder and decoder or "drift" when there are packet losses. These mismatches lead to a significant degradation in the decoded quality. To address this problem, we propose an auxiliary codec system that sends additional information alongside an MPEG or H.26x compressed video stream to correct for errors in decoded frames and mitigate drift. The proposed system is based on the principles of distributed source coding and uses the (possibly erroneous) MPEG/H.26x decoder reconstruction as side information at the auxiliary decoder. The distributed source coding framework depends upon knowing the statistical dependency (or correlation) between the source and the side information. We propose a recursive algorithm to analytically track the correlation between the original source frame and the erroneous MPEG/H.26x decoded frame. Finally, we propose a rate-distortion optimization scheme to allocate the rate used by the auxiliary encoder among the encoding blocks within a video frame. We implement the proposed system and present extensive simulation results that demonstrate significant gains in performance both visually and objectively (on the order of 2 dB in PSNR over forward error correction based solutions and 1.5 dB in PSNR over intrarefresh based solutions for typical scenarios) under tight latency constraints.

We describe PRISM, a video coding paradigm based on the principles of lossy distributed compression (also called source coding with side information or Wyner-Ziv coding) from multiuser information theory. PRISM represents a major departure from conventional video coding architectures (e.g., the MPEGx, H.26x families) that are based on motion-compensated predictive coding, with the goal of addressing some of their architectural limitations. PRISM allows for two key architectural enhancements: (1) inbuilt robustness to "drift" between encoder and decoder and (2) the feasibility of a flexible distribution of computational complexity between encoder and decoder. Specifically, PRISM enables transfer of the computationally expensive video encoder motion-search module to the video decoder. Based on this capability, we consider an instance of PRISM corresponding to a near reversal in codec complexities with respect to today's codecs (leading to a novel light encoder and heavy decoder paradigm), in this paper. We present encouraging preliminary results on real-world video sequences, particularly in the realm of transmission losses, where PRISM exhibits the characteristic of rapid recovery, in contrast to contemporary codecs. This renders PRISM as an attractive candidate for wireless video applications.

We study the problem of enhancing a baseline standards-compliant video-over-wireless broadcast system through the use of an auxiliary wireless broadcast channel. Our proposed solution is based on the information-theoretic concept of broadcast source coding which subsumes the concept of source coding with side-information. We integrate into this framework an analytically tractable mechanism for dynamically incorporating channel loss. Empirical validation of our proposed algorithm on a typical 3G wireless network broadcast channel simulator reveals its superior performance over pure FEC-based solutions by an average of 2-4 dB in PSNR.

In this work, we analytically study the complexity-performance trade-offs associated with video codecs based on the principle of source coding with side information at the decoder. We address three important aspects. First, we quantify the theoretical performance gains attained with side-information based codecs over prediction-based coders like MPEG under a lossy transmission scenario when there is drift. Secondly, we show that it is possible to closely approach MPEG's compression performance using side-information video coding principles with accurate DFD (displaced frame difference) modeling even without sophisticated channel codes. Thirdly, we analytically show that the value of accurately estimating DFD statistics diminishes as the channel gets noisier.

We describe PRISM (power-efficient, robust, high-compression, syndrome-based multimedia coding), an error-resilient video-coding paradigm built on the principles of distributed source coding from multi-user information theory. PRISM represents a radical departure from the current state-of-the-art video coding architectures, like MPEG, that are based on a motion-compensated prediction framework. These are hampered by: (i) a rigid computational complexity partition between encoder (heavy) and decoder (light); and (U) high fragility to drift between encoder and decoder in the face of prediction mismatch due to channel loss. In contrast, PRISM's architectural goals are: (i) to have a channel-adaptive distribution of computational complexity between encoder and decoder; and (ii) to have built-in robustness to drift between encoder and decoder due to wireless channel loss. These features make PRISM ideally suited for low-latency multimedia transmission over wireless networks, particularly for uplink-rich applications.

We present an algorithm for robust scalable video multicast based on distributed source coding techniques. Unlike prediction based coders, like MPEG, the proposed framework directly addresses the problem of drift due to packet losses. Building on the recently proposed PRISM video coding framework (R. Puri and K. Ramehandran, 2002), we show that substantial gains are possible for video multicast over lossy channels as compared to standard codecs, without a dramatic increase in encoder complexity as the number of streams increases.

Predictive video codecs such as MPEG and H.26x are very susceptible to prediction mismatch between encoder and decoder or "drift" when there are packet losses, leading to a significant reduction in the decoded quality. In this paper, we propose a method to reduce the drift in these codecs by sending extra information over a side-channel. Using principles from distributed source coding, we send another description of the video sequence (at a lower rate) over the side-channel which can correct for errors in the decoded frame and thereby mitigate the drift. Simulation results using our techniques show significant gains in performance over conventional error protection schemes such as Forward Error Correction codes under reasonable latency constraints.

We address the problem of streaming video efficiently from multiple servers in a distributed manner. Invoking a scalable video encoding format, we describe a network-friendly, rate-distortion efficient distributed streaming algorithm based on a robust distributed encoding paradigm. The algorithm is efficiently matched dynamically to both the video content and the network channel conditions. Due to the diversity effect of multiple servers, our distributed algorithm is immune to single points of failure, provides natural load-balancing of the multiple servers and a graceful degradation in quality with an increase in multi-server load or a decrease in network throughput.

We address the problem of streaming video efficiently from mUl­ tiple servers in a distributed manner. Invoking a scalable video encoding format, we describe a network-friendly , rate-distortion efficient distributed streaming algorithm based on a robust dis­ tributed encoding paradigm. The algorithm is efficiently matched dynamically to both the video content and the network channel conditions. Due to t he d iversity effect of multiple servers, our dis­ tributed algorithm is immune to single points of failure, provides natural load-balancing of the multiple servers and a graceful degra­ dation in quality with an Increase in multi-server load or a decrease in network throughput. We present the results of our Internet ex­ periments that validate our hypotheses.

We address the problem of real-time video streaming over wireless LANs for both unicast and multicast transmission. The wireless channel is modeled as a packet-erasure channel at the IP level. For the unicast scenario, we describe a novel hybrid Automatic Repeat reQuest (ARQ) algorithm that efficiently combines forward error control (FEC) coding with the ARQ protocol. For the multiple-users scenario, we formulate the problem of real-time video multicast as an optimization of a maximum regret cost function across the multicast user space. The proposed solution efficiently combines progressive source coding with FEC coding. We present a theoretical analysis of the unicast and multicast cases, as well as experimental results that demonstrate the performance advantages of the proposed algorithms over existing methods.

Orthogonal information present in the video signal associated with the audio helps in improving the accuracy of a speech recognition system. Audio-visual speech recognition involves extraction of both the audio as well as visual features from the input signal. Extraction of visual parameters is done by the recognition of speech dependent features from the video sequence. The paper uses geometrical features to describe the lip shapes. Curve-based active shape models are used to extract the geometry. These geometrically represented visual parameters are used along with the audio cepstral features to perform an audio-visual classification. It is shown that the bimodal system presented gives an improvement in the classification results over classification using only the audio features.