Communication has seen enormous advances over the past 100 years including radio, television, mobile phones, video conferencing, and Internet-based voice and video calling. Still, remote communication remains less natural and more fatiguing than face-to-face. The vision of immersive communication is to enable natural experiences and interactions with remote people and environments in ways that suspend disbelief in being there. This paper briefly describes the current state-of-the-art of immersive communication, provides a vision of the future and the associated benefits, and considers the technical challenges in achieving that vision. The attributes of immersive communication are described, together with the frontiers of video and audio for achieving them. We emphasize that the success of these systems must be judged by their impact on the people who use them. Recent high-quality video conferencing systems are beginning to deliver a natural experience-when all participants are in custom-designed studios. Ongoing research aims to extend the experience to a broader range of environments. Augmented reality has the potential to make remote communication even better than being physically present. Future natural and effective immersive experiences will be created by drawing upon intertwined research areas including multimedia signal processing, computer vision, graphics, networking, sensors, displays and sound reproduction systems, haptics, and perceptual modeling and psychophysics.

In this panel, each panelist will present their view of the current state-of-the-art of research and product innovations in the three major areas of multimedia experience: visual, auditory and gaming. We will discuss examples of innovation that enhance the consumption and sharing of multimedia (video, audio, graphics etc.) and thus increase quality of user experience. Another major focus of this panel is to open the discussion on how to innovate new multimedia user experiences.

The growing popularity of multimedia streaming applications brings a growth in diversity of media clients (laptops, PDAs, cellphones). Effectively serving this heterogeneous group of users is highly desirable. Scalable media codecs such as H.264/MPEG-4 SVC help make this adaptation possible. To account for the various capabilities and requests of each user, such as varying spatial or temporal resolutions, multiple distortion measures (MDM) are considered. Rather than consider a homogeneity in users, the MDM framework considers multiple different distortion values for each media packet for each user type. We consider the scenario of simultaneously broadcasting a video stream to multiple users over wireless links. The objective is to design a scheduling algorithm which achieves the highest aggregate quality-of-service, measured by distortion and delay, over all different user types. We cast the problem as a stochastic shortest path problem and use dynamic programming to find the optimal policy. For statistically static channels, the optimal policy is shown to be of threshold type. For time-varying channels, a quasi-static policy is introduced. Experimental results show that our policy reduces distortion by up to a factor of 2 over conventional approaches which do not consider MDM.

The growing diversity in media clients calls for content providers to adapt media content to adhere to their various needs. It is desirable to serve these heterogeneous users in a fast and efficient manner. Scalable media, such as H.264/MPEG- 4 SVC, helps make this possible. To account for various viewing capabilities of each user, such as different spatial or temporal resolutions, the Multiple Distortion Measures framework is used [1], [2]. MDM associates multiple distortion values with each packet depending on the user types (low/high resolution viewers, low/high frame rate viewers, etc.) who will consume the media packets. In this paper, we examine how to broadcast media packets with multiple distortion measures to multiple users. The tradeoff between media distortion and delay (in the form of retransmissions) plays an integral role in the scheduling decision. We cast the problem as a stochastic shortest path problem and use Dynamic Programming to find the optimal policy. In the case of statistically static channels, the optimal policy is shown to be a threshold policy where the number of allowable retransmissions is dictated by the importance, in terms of incurred distortion, of each packet. Through experimental results, we show that our policy, which considers multiple distortion measures, achieves up to 8 dB gains over conventional approaches. Finally, a policy based on the theoretical results of statistically static channels is empirically shown to have high performance for time-varying channels modeled by a two-state Markov Chain.

With the recent growth in mobile device types, an interesting question that arises is how to serve media content to users with a diverse set of capabilities. Defining multiple distortion metrics, one for each type of user, allows for specialized service to each user type, which can result in significantly improved quality of service. H.264/MPEG-4 SVC is a scalable video codec that allows for easy spatial and temporal adaptation of encoded video by selecting or dropping packets. We examine using multiple distortion measures (MDM) for scheduling packets in the context of streaming temporally scalable video to users with different target frame rates. We show that gains in PSNR of multiple dB can be achieved when taking MDM into consideration for making scheduling decisions.

As the diversity in end-user devices and networks grows, it becomes important to be able to efficiently and adaptively serve media content to different types of users. A key question surrounding adaptive media is how to do Rate-Distortion optimized scheduling. Typically, distortion is measured with a single distortion measure, such as the Mean-Squared Error compared to the original high resolution image or video sequence. Due to the growing diversity of users with varying capabilities such as different display sizes and resolutions, we introduceMultipleDistortionMeasures(MDM) to account for a diverse range of users and target devices. MDM gives a clear framework with which to evaluate the performance of media systems which serve a variety of users. Scalable coders, such as JPEG2000 and H.264/MPEG-4 SVC, allow for adaptation to be performed with relatively low computational cost. We show that accounting for MDM can significantly improve system performance; furthermore, by combining this with scalable coding, this can be done efficiently. Given these MDM, we propose an algorithm to generateembeddedschedules, which enables low-complexity, adaptive streaming of scalable media packets to minimize distortion across multiple users. We show that using MDM achieves up to 4 dB gains for spatial scalability applied to images and 12 dB gains for temporal scalability applied to video.

The increasing diversity of end-user devices and networks allow different users to receive and view images and video at different resolutions and rates. Scalable coding methods allow streaming media systems to easily adapt media by selecting packets according to schedules that reflect their importance. Traditional scheduling algorithms are based on a single distortion measure such as mean-squared error relative to the original high resolution image. In this paper, we present a new approach of using multiple distortion measures to schedule packets in a manner that explicitly accounts for a range of rates and resolutions. We show the effectiveness of this approach and examine a common scenario where multiple distortion measures are helpful. We present an algorithm that generates embedded JPEG2000 schedules and achieves 1 to 4 dB improvement over conventional approaches.

Video streaming with authentication is practically important, where a packet is decoded only when it is both received and authenticated. Recent work examined the problem of rate-distortion-authentication (R-D-A) optimized streaming of authenticated video. The original R-D-A technique assumes that each packet has only one deadline, its display deadline, and that a packet is not considered for transmission after its deadline. However, for video protected with an inter-packet graph-based authentication technique, a video packet can still be useful for verification of other packets even if it misses its own display deadline. We formulate the problem of multiple-deadline R-D-A optimized streaming and also propose ways to reduce the complexity. Simulation results using H.264 and NS-2 demonstrate that multiple-deadline R-D-A optimization achieves performance improvements of up to 4 dB over single-deadline R-D-A optimization.

This paper proposes a stream authentication method based on the generalized butterfly graph (GBG) framework. Compared with the original Butterfly graph, the proposed GBG graph supports an arbitrary overhead budget and number of packets. Within the GBG framework, the problem of constructing an authentication graph is considered as a design problem: Given total number of packets, packet loss rate, and overhead budget, we show how to design the graph (number of rows and columns and edge allocation among nodes) to maximize the expected number of verified packets. In addition, we also propose a new evaluation metric called loss-amplification-factor (LAF), which measures the extent to which the authentication method exacerbates the effective packet loss rate. Experimental results demonstrate significant performance improvements over existing authentication methods like EMSS, augmented chain, and the original Butterfly.

Due to the increase in diversity of wireless devices, streaming media systems must be capable of serving multiple types of users. Scalable coding allows for adaptations without re-encoding. To account for various viewing capabilities of each user, such as different spatial resolutions, multiple distortion measures are used. In this paper, we examine the question of how to broadcast media packets with multiple distortion measures to multiple users. We cast the problem as a stochastic shortest path problem and use Dynamic Programming to find the optimal policy. We generate an offline algorithm to generate the optimal transmission policy for the general case. We then show the optimal policy can be done online via a simple threshold policy for the case of independent Bernoulli packet losses. Through experimental results, we show that our policy, which considers multiple distortion measures, achieves up to 2dB gains over conventional approaches.

We define authenticated video as decoded video that results from those received packets whose authenticities have been verified. Generic data stream authentication methods usually impose overhead and dependency among packets for verification. Therefore, the conventional rate-distortion (R-D) optimized video streaming techniques produce highly sub-optimal R-D performance for authenticated video, since they do not account for the overhead and additional dependencies for authentication. In this paper, we study this practical problem and propose an Rate-Distortion-Authentication (R-D-A) optimized streaming technique for authenticated video. Based on packets' importance in terms of both video quality and authentication dependencies, the proposed technique computes a packet transmission schedule that minimizes the expected end-to-end distortion of the authenticated video at the receiver subject to a constraint on the average transmission rate. Simulation results based on H.264 JM 10.2 and NS-2 demonstrate that our proposed R-D-A optimized streaming technique substantially outperforms both prior (authentication-unaware) R-D optimized streaming techniques and data stream authentication techniques. In particular, when the channel capacity is below the source rate, the PSNR of authenticated video quickly drops to unacceptable levels using conventional R-D optimized streaming techniques, while the proposed R-D-A Optimization technique still maintains optimized video quality. Furthermore, we examine a low-complexity version of the proposed algorithm, and also an enhanced version which accounts for the multiple deadlines associated with each packet, which is introduced by stream authentication

Many algorithmic efforts have been made to address technical issues in designing a streaming media caching proxy. Typical of those are segment-based caching approaches that efficiently cache large media objects in segments which reduces the startup latency while ensuring continuous streaming. However, few systems have been practically implemented and deployed. The implementation and deployment efforts are hindered by several factors: 1) streaming of media content in complicated data formats is difficult; 2) typical streaming protocols such as RTP often run on UDP; in practice, UDP traffic is likely to be blocked by firewalls at the client side due to security considerations; and 3) coordination between caching discrete object segments and streaming continuous media data is challenging. To address these problems, we have designed and implemented a segment-based streaming media proxy, called SProxy. This proxy system has the following merits. First, SProxy leverages existing Internet infrastructure to address the flash crowd. The content server is now free of the streaming duty while hosting streaming content through a regular Web server. Thus, UDP based streaming traffic from SProxy suffers less dropping and no blocking. Second, SProxy streams and caches media objects in small segments determined by the object popularity, causing very low startup latency, and significantly reducing network traffic. Finally, prefetching techniques are used to pro-actively preload uncached segments that are likely to be used soon, thus providing continuous streaming. SProxy has been extensively tested and we show that it provides high quality streaming delivery in both local area networks and wide area networks (e.g., between Japan and the U.S.).

This paper proposes an optimized content-aware authentication scheme for JPEG-2000 streams over lossy networks, where a received packet is consumed only when it is both decodable and authenticated. In a JPEG-2000 codestream, some packets are more important than others in terms of coding dependency and image quality. This naturally motivates allocating more redundant authentication information for the more important packets in order to maximize their probability of authentication and thereby minimize the distortion at the receiver. Towards this goal, with the awareness of its corresponding image content, we formulate an optimization framework to compute an authentication graph to maximize the expected media quality at the receiver, given specific authentication overhead and knowledge of network loss rate. System analysis and experimental results demonstrate that the proposed scheme achieves our design goal in that the rate-distortion (R-D) curve of the authenticated image is very close to the R-D curve when no authentication is required

In this paper, we propose an optimized content-aware authentication scheme for JPEG-2000 streams over lossy networks, where a received packet is consumed only when it is both decodable and authentic. In a JPEG-2000 codestream some packets are more important than others in terms of coding dependency and visual quality. This inspires us to allocate more redundant authentication information for the more important packets to minimize the distortion of the authenticated image at the receiver. In other words, with the awareness of image content, we formulate an optimization framework, which is able to build an authentication graph yielding the best visual quality at the receiver, given a specific authentication overhead and network condition. Experimental results demonstrate that the proposed scheme achieved our design goals in that the R-D curve of an authenticated image is very close to its original one where no authentication is applied.

The JPEG-2000 compression standard codes images into data units, referred to as packets, such that images can be successfully decoded, while incurring some distortion, with a subset of these packets. This paper examines how to optimally select subsets of packets (referred to as schedules) that minimize distortion subject to varying rate constraints. We solve for the optimal schedule at a single rate by solving the precedence constraint knapsack problem (PCKP) via dynamic programming, and show that with modifications that consider the specific dependencies of JPEG-2000 packets, we can compute the optimal schedules for all rates through a single execution of our algorithm. We then analyze important properties of the optimal schedule. Using these properties, we look at a fused-greedy algorithm, similar to the recently proposed convex hull algorithm, to generate embedded schedules of JPEG-2000 packets. These embedded schedules have the property that all the JPEG-2000 packets in lower rate schedules are included in higher rate schedules; embedded schedules enable low-complexity, adaptive streaming. We demonstrate the algorithm's near optional performance through comparisons to the optimal performance for JPEG-2000 coded data.

Stream authentication methods usually impose overhead and dependency among packets. The straightforward application of state-of-the-art rate-distortion (R-D) optimized streaming techniques produce highly sub-optimal R-D performance for authenticated video, since they do not account for the additional dependencies. This paper proposes an R-D optimized streaming technique for authenticated video, by accounting for authentication dependencies and overhead. It schedules packet transmission based on packets' importance in terms of both video quality and authentication dependencies. The proposed technique works with any stream authentication method as long as the verification probability can be quantitatively computed from packet loss probability. Simulation results based on H.264 JM 10.1 and NS-2 demonstrate that the proposed authentication-aware R-D optimized streaming technique substantially outperforms authentication-unaware R-D optimized streaming techniques. In particular, when the channel capacity is below the source rate, the PSNR of authenticated video quickly drops to unacceptable levels using conventional R-D optimized streaming techniques, while the proposed technique still maintains R-D optimized video quality.

Hosting Internet streaming services has its unique challenges. Aiming at making Internet streaming services be widely and easily adopted in practice, in this paper, we have designed and implemented a system, called SProxy that can leverage existing Internet infrastructure to free the streaming content providers so that they only need to host streaming content through a regular Web server. SProxy has been extensively tested and evaluated and it provides high quality streaming delivery in both local area networks and wide area networks (e.g. between Japan and US)

This paper proposes a content-aware authentication scheme optimized to account for distortion and overhead for media streaming. When authenticated media is streamed over a lossy network, a received packet is consumed only when it is both decodable and authenticated. In most media formats, some packets are more important than others. This naturally motivates allocating more redundant authentication information for the more important packets in order to maximize their probability of authentication and thereby minimize distortion at the receiver. Toward this goal, with awareness of the media content, we formulate an optimization framework to compute an authentication graph to maximize the expected media quality at the receiver, given specific authentication overhead and knowledge of network loss rates. Experimental results with JPEG-2000 coded images demonstrate that the proposed method achieves our design goal in that the R-D curve of the authenticated image is very close to the R-D curve when no authentication is required

The emergence of digital imaging applications is accelerating the need for security of digital imagery. The emerging international standard ISO/IEC JPEG-2000 security (JPSEC) is designed to provide security for digital imagery, and in particular digital imagery coded with the JPEG-2000 image coding standard. This paper provides an overview of the JPSEC standard, including a description of its basic architecture and examples of its use.

Researchers often use segment-based proxy caching strategies to deliver streaming media by partially caching media objects. The existing strategies mainly consider increasing the byte hit ratio and/or reducing the client perceived startup latency (denoted by the metric delayed startup ratio). However, these efforts do not guarantee continuous media delivery because the to-be-viewed object segments may not be cached in the proxy when they are demanded. The potential consequence is playback jitter at the client side due to proxy delay in fetching the uncached segments, which we call proxy jitter. Thus, for the best interests of clients, a correct model for streaming proxy system design should aim to minimize proxy jitter subject to reducing the delayed startup ratio and increasing the byte hit ratio. However, we have observed two major pairs of conflicting interests inherent in this model: (1) one between improving the byte hit ratio and reducing proxy jitter, and (2) the other between improving the byte hit ratio and reducing the delayed startup ratio. In this study, we first propose and analyze prefetching methods for in-time prefetching of uncached segments, which provides insights into the first pair of conflicting interests. Second, to address the second pair of the conflicting interests, we build a general model to analyze the performance tradeoff between the second pair of conflicting performance objectives. Finally, considering our main objective of minimizing proxy jitter and optimizing the two tradeoffs, we propose a new streaming proxy system called Hyper Proxy. Synthetic and real workloads are used to evaluate our system. The performance results show that Hyper Proxy generates minimum proxy jitter with a low delayed startup ratio and a small decrease of byte hit ratio compared with existing schemes.

We present a technique for low-complexity rate-distortion (R-D) optimized adaptive video streaming based on the concept of rate-distortion hint track (RDHT). RDHTs store the precomputed characteristics of a compressed media source that are crucial for high performance online streaming but difficult to compute in real time. This enables low-complexity adaptation to variations in transport conditions such as available data rate or packet loss. An RDHT-based streaming system has three components: 1) information that summarizes the R-D attributes of the media; 2) an algorithm for using the RDHT to predict the distortion for a feasible packet schedule; and 3) a method for determining the best packet schedule to adapt the streaming to the communication channel. A family of distortion models, denoted distortion chains, are presented which accurately predict the distortion produced by arbitrary packet loss patterns. Two distortion chain models are examined which lead to two RDHT-based techniques. We evaluate the proposed techniques for two canonical problems in streaming media, adaptation to available data rate and to packet loss. Experimental results demonstrate that for the difficult case of nonscalably coded H.264 video, the proposed systems provide significant performance gains over conventional low-complexity streaming systems, and achieve this gain with a comparable level of complexity making them suitable for online R-D optimized streaming.

When designing a system for video communication over a lossy packet network, it is highly beneficial to have a mechanism for accurately predicting the mean-squared error (MSE) distortion that results from different packet loss patterns. The paper proposes a distortion chains model for accurately predicting the end-to-end distortion for different general packet loss patterns. The performance is examined using JVT/H.264 encoded video sequences and previous frame error concealment. It is shown that, for all tested sequences, the proposed model predicts the total distortion due to a packet loss pattern within a 10% error bound 80% of the time, as compared to the conventional additive approach which achieves the same accuracy less then 40% of the time.

The emerging JPEG-2000 part 8 security standard (JPSEC) is being defined to provide security services for JPEG-2000 images. This paper describes how confidentiality and authentication can be applied in a manner that allows mid-network adaptation of protected JPSEC streams while preserving end-to-end security. We achieved this by designing the JPSEC syntax to support the principles of secure scalable streaming and secure transcoding. Specifically, we designed JPSEC encryption methods and signaling syntax that enable an entity to securely adapt or transcode the resulting JPSEC-protected stream without requiring decryption. We discuss tradeoffs in protection, transcoding flexibility, and complexity for the different encryption methods. Furthermore, we show how authentication can be applied to verify that the secure transcoding operation was performed in a valid and permissible manner.

This work presents the concept of rate-distortion hint track (RDHT), and evaluates two specific implementations of streaming systems that employ RDHT. Using RDHT, low-complexity streaming can be realized for systems that adapt to variations in transport conditions such as bandwidth or packet loss. An RDHT-based streaming system has three components: (1) an R-D hint track; (2) an algorithm for using the RDHT to predict the distortion for different packet schedules; and (3) a method for determining the best packet schedule. Two RDHT-based systems are presented which perform R-D optimized scheduling with dramatically reduced complexity as compared to conventional on-line R-D optimized streaming algorithms. Experimental results demonstrate that for the difficult case of R-D optimized scheduling of non-scalably coded video the proposed systems provide 7-12 dB gain when adapting to a bandwidth constraint and 2-4 dB gain when adapting to random packet loss, both relative to a conventional streaming system that does not take into account the different importance of individual packets.

Recent years have witnessed significant efforts in deriving and embedding semantic information in content for improved content retrieval, adaptation, and distribution. Relatively little work considers leveraging this semantic information in the infrastructure to better serve the needs of clients and achieve better cost-effectiveness of infrastructure resources. In this paper, we identify semantics that can be derived and extracted from the various components in a content distribution infrastructure, namely content semantics (from content source), infrastructure semantics and client semantics (from content consumer). We develop a semantic-enhanced distribution and adaptation framework (SEDAN) that achieves superior efficiency and provides content access and adaptation features that were not previously possible

Recent years have witnessed significant efforts in deriving and embedding semantic information in content for improved content retrieval, adaptation, and distribution. Relatively little work considers leveraging this semantic information in the infrastructure to serve the needs of clients better and achieve better cost-effectiveness of infrastructure resources. We identify semantics that can be derived and extracted from the various components in a content distribution infrastructure, namely content semantics (from content source), infrastructure semantics and client semantics (from content consumer). We develop a semantic-enhanced distribution and adaptation framework (SEDAN) that achieves superior efficiency and provides content access and adaptation features that were not previously possible.

Researchers often use segment-based proxy caching strategies to deliver streaming media by partially caching media objects. The existing strategies mainly consider increasing the byte hit ratio and/or reducing the client perceived startup latency (denoted by the metric delayed startup ratio). However, these efforts do not guarantee continuous media delivery because the to-be-viewed object segments may not be cached in the proxy when they are demanded. The potential consequence is playback jitter at the client side due to proxy delay in fetching the uncached segments, which we call proxy jitter. Thus, for the best interests of clients, a correct model for streaming proxy system design should aim to minimize proxy jitter subject to reducing the delayed startup ratio and increasing the byte hit ratio. However, we have observed two major pairs of conflicting interests inherent in this model: (1) one between improving the byte hit ratio and reducing proxy jitter, and (2) the other between improving the byte hit ratio and reducing the delayed startup ratio. In this study, we first propose an active prefetching method for in-time prefetching of uncached segments, which provides insights into the first pair of conflicting interests. Second, we further improve our lazy-segmentation scheme which effectively addresses the second pair of the conflicting interests. Finally, considering our main objective of minimizing proxy jitter and optimizing the two trade-offs, we propose a new streaming proxy system called Hyper Proxy by effectively coordinating both prefetching and segmentation techniques. Synthetic and real workloads are used to systematically evaluate our system. The performance results show that the hyper proxy system generates minimum proxy jitter with a low delayed startup ratio and a small decrease of byte hit ratio compared with existing schemes.

Diversity techniques often arise as appealing means for improving the performance of multimedia communication over certain types of channels with independent parallel components (e.g., multiple antennas, frequency bands or time slots). Diversity can be obtained by channel coding across parallel components at the physical layer. Alternatively, the physical layer ca present an interface to the parallel components as separate, independent links thus allowing the application layer to implement diversity in the form of multiple description source coding. We compare these two approaches in terms of average end-to-end distortion as a function of channel signal-to-noise ratio (SNR). When specialized to the case of an independent, identically distributed Gaussian source over Rayleigh fading channels, our results suggest that parallel channel coding at the physical layer is more efficient than independent channel coding combined with multiple description source coding. More generally, we provide intuitive guidelines for allowing system designers to identify which types of systems are preferable under different scenarios of practical interest.

Secure scalable streaming (SSS) enables low-complexity, high-quality transcoding at intermediate, possibly untrusted, network nodes without compromising the end-to-end security of the system S. J. Wee, J. G. Apostolopoulos (2001). SSS encodes, encrypts, and packetizes video into secure scalable packets in a manner that allows downstream transcoders to perform transcoding operations such as bitrate reduction and spatial downsampling by simply truncating or discarding packets, and without decrypting the data. Secure scalable packets have unencrypted headers that provide hints such as optimal truncation points to downstream transcoders. Using these hints, downstream transcoders can perform near-optimal secure transcoding. This paper presents a secure scalable streaming system based on motion JPEG-2000 coding with AES or triple-DES encryption. The operational rate-distortion (R-D) performance for transcoding to various resolutions and quality levels is evaluated, and results indicate that end-to-end security and secure transcoding can be achieved with near R-D optimal performance. The average overhead is 4.5% for triple-DES encryption and 7% for AES, as compared to the original media coding rate, and only 2-2.5% overhead as compared to end-to-end encryption which does not allow secure transcoding.

We present MSA (media services architecture), a flexible, general architecture for requesting, configuring, and running services that operate on streaming audio and video as it flows through the network. MSA decomposes requested media services into modular processing components that may be distributed to servers throughout the network and which intercommunicate via standard streaming protocols. Use of standard protocols also affords seamless interoperability between MSA and media content delivery networks. MSA manages media services by monitoring the networked servers and assigning service components to them in a manner that uses available computational and network resources efficiently. We describe some implemented example services to illustrate the concepts and benefits of the architecture.

A mobile streaming media content delivery network can facilitate the access of rich multimedia streams by mobile users on next generation wireless networks. A principal component of this network is the edge servers that cooperate to provide streaming functionalities. This paper describes some of the design requirements for such servers. It then describes the architecture of such a streaming content, a modular design, and an intelligent scheduler for providing better video quality in varying network conditions. Finally, preliminary performance results are shown.

Delivering media to large numbers of mobile users presents challenges due to the stringent requirements of streaming media, mobility, wireless, and scaling to support large numbers of users. This paper presents a mobile streaming media content delivery network (MSM-CDN) designed to overcome these challenges. The MSM-CDN is a network overlay consisting of overlay servers on top of the existing network; these overlay servers are control points that facilitate end-to-end media delivery and mid-network media services. This paper presents an overview of the MSM-CDN system architecture, and describes the testbed prototype that we built based on these architectural principles. The MSM-CDN provides a new platform for media delivery, and we describe a number of research directions related to the MSM-CDN.

Streaming media objects are often cached in segments. Previous segment-based caching strategies cache segments with constant or exponentially increasing lengths and typically favor caching the beginning segments of media objects. However, these strategies typically do not consider the fact that most accesses are targeted toward a few popular objects. In this paper, we argue that neither the use of a predefined segment length nor the favorable caching of the beginning segments is the best caching strategy for reducing network traffic. We propose an adaptive and lazy segmentation based caching mechanism by delaying the segmentation as late as possible and determining the segment length based on the client access behaviors in real time. In addition, the admission and eviction of segments are carried out adaptively based on an accurate utility function. The proposed method is evaluated by simulations using traces including one from actual enterprise server logs. Simulation results indicate that our proposed method achieves a 30% reduction in network traffic. The utility functions of the replacement policy are also evaluated with different variations to show its accuracy.

The use of multiple description (MD) video coding and path diversity has been proposed to provide improved performance over lossy packet networks [1]. The goal of this work was to develop models to accurately and quickly predict and compare the distortion of MD video coding and path diversity against conventional single description (SD) video delivered over a single path. In the process, we developed (1) a model for the loss process of a two-path path diversity system, and (2) a distortion model that maps the loss model to MD distortion values. Given these models we present a number of comparisons between MD video coding and path diversity and conventional SD video over a single path. The proposed model for path diversity may also be useful in other applications not related to MD coding. Furthermore, other forms of MD coding may be analyzed using similar models for MD distortion.

Content delivery networks (CDN) have been widely used to provide reduced delay and packet loss, fault tolerance, and improved scalability for Web content delivery. Additional benefits are provided for video streaming when one designs a streaming media CDN (SM-CDN) for either conventional single description (SD) or multiple description (MD) coding. Specifically, when precise network conditions and topology are known, simulations show that an MD-SM-CDN can provide 20 to 40% reduction in distortion over a conventional SD-SM-CDN, even when the underlying CDN is not designed with MD streaming in mind. This paper examines the performance of an MD-SM-CDN as a function of different network topologies and loss conditions, and compares it with a conventional SD-SM-CDN. This examination provides insight into an MD-SM-CDN's performance when knowledge of network topology and conditions is imprecise or uncertain. Our simulations indicate that an MD-SM-CDN can provide improved performance over a conventional SD-SM-CDN over a wide range of network topologies and loss conditions.

This paper presents a method for distortion-optimized streaming of predictively coded video over packet networks with varying delay. In networks with significant delay variations, coded video frames can arrive late at the decoder and miss their respective display deadlines. Furthermore, due to predictive coding, a late frame can also prevent a number of subsequent frames from being displayed properly, where the number of affected frames or degree of distortion depends on the particular coding dependencies of the late frame. In this paper, we present an optimized video streaming strategy based on frame reordering for networks with significant delay variations. This streaming strategy minimizes distortion by exploiting the fact that different late frames result in different degrees of distortion. We model the router-induced delay in a wired network with an analytical PDF and we model the link-layer retransmission delay of a wireless network with the 3GPP specification for W-CDMA radio link control. We compute the distortion for different frame reorderings using the network delay models and a source model that accounts for the prediction dependencies of predictively coded video. Our optimized streaming strategies are shown to reduce the number of late frames by 14 to 23% for the situations examined.

We propose a system that improves the performance of streaming media CDN by exploiting the path diversity provided by existing CDN infrastructure. Path diversity is provided by the different network paths that exist between a client and its nearby edge servers; and multiple description (MD) coding is coupled with this path diversity to provide resilience to losses. In our system, MD coding is used to code a media stream into multiple complementary descriptions, which are distributed across the edge servers in the CDN. When a client requests a media stream, it is directed to multiple nearby servers which host complementary descriptions. These servers simultaneously stream these complementary descriptions to the client over different network paths. This paper provides distortion models for MDC video and conventional video. We use these models to select the optimal pair of servers with complementary descriptions for each client while accounting for path lengths and path jointness and disjointness. We also use these models to evaluate the performance of MD streaming over CDN in a number of real and generated network topologies. Our results show that distortion reduction by about 20 to 40% can be realized even when the underlying CDN is not designed with MDC streaming in mind. Also, for certain topologies, MDC requires about 50% fewer CDN servers than conventional streaming techniques to achieve the same distortion at the clients.

In this paper, we present a mobile streaming media CDN (Content Delivery Network) architecture in which content segmentation, request routing, pre-fetch scheduling, and session handoff are controlled by SMIL (Synchronized Multimedia Integrated Language) modification. In this architecture, mobile clients simply follow modified SMIL files downloaded from a streaming portal server; these modifications enable multimedia content to be delivered to the mobile clients from the best surrogates in the CDN. The key components of this architecture are 1) content segmentation with SMIL modification, 2) on-demand rewriting of URLs in SMIL, 3) pre-fetch scheduling based on timing information derived from SMIL, 4) SMIL updates by SOAP (Simple Object Access Protocol) messaging for session handoffs due to clients mobility. We also introduce QoS control with a network called an RTP monitoring agent to enable appropriate control of media quality based on both network congestion and radio link conditions. The current status of our prototyping on a mobile QoS testbed MOBIQ is reported in this paper. We are currently designing the SOAP-based APIs (Application Programmable Interfaces) needed for the mobile streaming media CDN and building the CDN over the current testbed.

We present a wireless video streaming system that securely and efficiently streams video to heterogeneous clients over timevarying communication links. Clients may differ in their display, power, communication, and computational capabilities and wireless channels may have time-varying bandwidths and quality levels that depend on channel usage and channel conditions. End-to-end system efficiency is achieved by placing transcoders at intermediate network nodes; these transcoders can easily adapt the video stream for particular client capabilities and network conditions.

We present a method of secure scalable streaming (SSS) that enables low-complexity and high-quality transcoding to be performed at intermediate, possibly untrusted, network nodes without compromising the end-to-end security of the system. SSS encodes video into secure scalable packets using jointly designed scalable coding and progressive encryption techniques. This combination allows downstream transcoders to perform transcoding operations such as bitrate reduction and spatial downsampling by simply truncating or discarding packets, and without decrypting the data. Secure scalable packets have unencrypted headers that can provide hints such as optimal truncation points to downstream transcoders. Using these hints, downstream transcoders can perform RD-optimal transcoding for fine-grain bitrate reduction. The SSS transcoding operation has low complexity and is stateless, so SSS transcoders can support many simultaneous transcoding sessions. SSS works with existing scalable image and video compression standards and systems including Motion JPEG-2000, 3D subband coding, and MPEG-4 FGS.

Multiple description (MD) coders provide important error resilience properties. Specifically, MD coders are designed to provide good performance when the loss is limited to a single description, but it is not known in advance which description. Apostolopoulos (2001) combined MD video coding with a path diversity transmission system for packet networks such as the Internet, where different descriptions are explicitly transmitted through different network paths, to improve the effectiveness of MD coding over a packet network by increasing the likelihood that the loss probabilities for each description are independent. The available bandwidth in each path may be similar or different, resulting in the requirement of balanced or unbalanced operation, where the bit rate of each description may differ based on the available bandwidth along its path. We design a MD video communication system that is effective in both balanced and unbalanced operation. Specifically, unbalanced MD streams are created by carefully adjusting the frame rate of each description, thereby achieving unbalanced rates of almost 2:1 while preserving MD's effectiveness and error recovery capability.

The ability to manipulate the temporal dependencies in coded video data is important for a number of compressed domain video processing tasks. This paper formulates the general problem and examines it in the context of MPEG. This is used to develop a method for performing frame conversions in MPEG coded video data. These frame conversions are used to develop compressed-domain video processing algorithms for performing temporal mode conversion, frame-by-frame reverse play, and frame-accurate splicing.

We present an algorithm for transcoding high-rate compressed bitstreams containing field-coded interlaced video to lower-rate compressed bitstreams containing frame-coded progressive video. We focus on MPEG-2 to H.263 transcoding, however these results can be extended to other lower-rate video compression standards including MPEG-4 simple profile and MPEG-1. A conventional approach to the transcoding problem involves decoding the input bitstream, spatially and temporally downsampling the decoded frames, and re-encoding the result. The proposed transcoder achieves improved performance by exploiting the details of the MPEG-2 and H.263 compression standards when performing interlaced to progressive (or field to frame) conversion with spatial downsampling and frame-rate reduction. The transcoder reduces the MPEG-2 decoding requirements by temporally downsampling the data at the bitstream level and reduces the H.263 encoding requirements by largely bypassing H.263 motion estimation by reusing the motion vectors and coding modes given in the input bitstream. In software implementations, the proposed approach achieved a 5/spl times/ speedup over the conventional approach with only a 0.3 and 0.5 dB loss in PSNR for the Carousel and Bus sequences.

We examine the problem of estimating a reverse motion vector field from a given forward motion vector field and its motion-compensated residual. This problem has practical importance when providing reverse-play functionality in compressed video environments, where video data is stored, processed, and transported in compressed form. We present a family of algorithms that trade off motion vector accuracy for computational efficiency. Experimental results demonstrate the performance of these algorithms.

This paper investigates the problem of communicating video over a broadcast channel. The broadcast channel is expressed in terms of the channel capacity that exists between the transmitter and each receiver in the broadcast area-Shannon's separation theorem does not apply for video communication over this class of channels. Digital (discrete time, discrete amplitude) and hybrid (discrete time, discrete/continuous amplitude) transmission and video coding methods are discussed. Joint source and channel coding principles are employed to effectively couple these methods to form efficient systems for communicating video over broadcast channels. A framework is presented for characterizing and bounding the performance of these systems; the results lead to interesting directions for future work.

An algorithm is proposed for efficiently splicing two MPEG-coded video streams. The algorithm only processes the portion of each video stream that contains frames affected by the splice and it operates directly on the DCT coefficients and motion vectors of those frames. The algorithm achieves good visual performance with very low computational complexity, and incorporates rate control to prevent buffer underflow and overflow. If additional processing power is available, it can be used to further improve the quality of the spliced stream.

This paper describes a new HDTV system that applies joint multiresolution (MR) source and channel coding to efficiently use the available radio spectrum. Hybrid analog/digital MR channel modulation provides the benefits of digital source coding and the more efficient spectrum usage of analog transmission. Nonuniform spacing of the digital signaling levels provides MR delivery of the digital components while spread-spectrum processing permits MR delivery of the analog components. Error correction coding and OFDM channel modulation deliver the MR service in widely varying channel conditions. Simulation results demonstrate performance in various regions of the service area.>

A scalable source coder was designed for a hybrid HDTV terrestrial broadcasting system. The hybrid combination of analog and digital methods allows for both high video compression ratios and efficient utilization of the available channel capacity. Efficient channel utilization enables closer receivers to decode high quality video, and allows further receivers to decode natural looking, though lower quality video. The system uses joint source/channel coding to deliver different video components with degrees of integrity reflecting their perceptual importance. Motivation is given for hybrid transmission and analog/digital coding methods. Details of the scalable source coder are provided with a description of its key features, which include pyramid filtering and hybrid video coding. Furthermore, fundamental differences between conventional digital video compression schemes and the proposed scalable hybrid approach are highlighted. Finally, frames of video produced by a computer simulation of the hybrid HDTV system are shown, demonstrating the feasibility of such a system.>