In this work, we study a challenging research problem that arises in minimizing the cost of storing customer data online for reliable access in a cloud. It is how to near-perfectly balance the remaining capacities of all disks across the cloud system while adding new file blocks so that the inevitable event of capacity expansion can be postponed as much as possible. The challenges of solving this problem are twofold. First, new file blocks are added to the cloud concurrently by many dispatchers (computing servers) that have no communication or coordination among themselves. Though each dispatcher is updated with information on disk occupancies, the update is infrequent and not synchronized. Second, for fault-tolerance purposes, a combinatorial constraint has to be satisfied in distributing the blocks of each new file across the cloud system. We propose a randomized algorithm, in which each dispatcher independently samples a blocks-to-disks assignment according to a probability distribution on a set of assignments conforming to the aforementioned combinatorial requirement. We show that this algorithm allows a cloud system to near-perfectly balance the remaining disk capacities as rapidly as theoretically possible, when starting from any unbalanced state that is correctable mathematically.

In this work, we study a challenging research problem that arises in minimizing the cost of storing customer data online for reliable accesses in a cloud. It is how to near-perfectly balance the remaining capacities of all disks across the cloud system while adding new file blocks so that the inevitable event of capacity expansion can be postponed as much as possible. The challenges of solving this problem are twofold. First, new file blocks are added to the cloud concurrently by many dispatchers (computing servers) that have no communication or coordination among themselves. Though each dispatcher is updated with information on disk occupancies, the update is infrequent and not synchronized. Second, for fault-tolerance purposes, a combinatorial constraint has to be satisfied in distributing the blocks of each new file across the cloud system. We propose a randomized algorithm, in which each dispatcher independently samples a blocks-to-disks assignment according to a probability distribution on a set of assignments conforming to the aforementioned combinatorial requirement. We show that this algorithm allows a cloud system to near-perfectly balance the remaining disk capacities as rapidly as theoretically possible, when starting from any unbalanced state that is correctable mathematically.

Content Delivery Networks (CDNs) play a central role in today's Internet infrastructure and have seen a sharp increase in scale. More and more internet sites are armed with live contents, such as live sports game statistics, e-commerce, and online auctions, and they rely on CDNs to deliver such contents freshly at scale. However, the problem of maintaining consistency for live (dynamic) contents while achieving high scalability is non-trivial in CDNs. The large number of widely scattered replicas guarantees the QoS of end-users while substantially increasing the complexity of consistency maintenance under frequent updates. Current consistency maintenance infrastructures and methods cannot simultaneously satisfy both scalability and consistency. In this paper, we first analyze our crawled trace data of cached sports game content on thousands of content servers of a major CDN. We analyze the content consistency from different perspectives, from which we break down the reasons for inconsistency among content servers. We verify that the CDN uses unicast instead of multicast trees as the update infrastructure, which may not scale effectively. Then, we further evaluate the performance in consistency, scalability and overhead for different infrastructures with different update methods. We itemize the advantages and disadvantages of different methods and infrastructures in different scenarios through the evaluation. Based on this evaluation, we propose our hybrid and self-adaptive update method to reduce network load and improve scalability under the conditions recorded in the trace and prove its effectiveness through trace-driven experiments. We aim to give guidance for appropriate selections of consistency maintenance infrastructures and methods for a CDN, and for choosing a CDN service with different considerations.

With the advances in distributed computation, machine learn-ing and deep neural networks, we enter into an era that it is possible to build a real world image recognition system. There are three essential components to build a real-world image recognition system: 1) creating representative features, 2) de-signing powerful learning approaches, and 3) identifying massive training data. While extensive researches have been done on the first two aspects, much less attention has been paid on the third. In this paper, we present an end-to-end Web knowledge discovery system, Prajna. Starting from an arbi-trary set of entities as inputs, Prajna automatically crawls im-ages from multiple sources, identifies images that have relia-bly labeled, trains models and build a recognition system that is capable of recognizing any new images of the entity set. Due to the high cost of manual data labeling, leveraging the massive yet noisy data on the Internet is a natural idea, but the practical engineering aspect is highly challenging. Prajna fo-cuses on separating reliable training data from extensive noisy data, which is a key to the capability of extending an image recognition system to support arbitrary entities. In this paper, we will analyze the intrinsic characteristics of Internet image data, and find ways to mine accurate and informative infor-mation from those data to build a training set, which is then used to train image recognition models. Prajna is capable of automatically building an image recognition system for those entities as long as we can collect sufficient number of images of the entities on the Web.

With the rise of large-scale, Web-based applications, users are increasingly adopting a new class of document-oriented database management systems (DBMSs) that allow for rapid prototyping while also achieving scalable performance. Like for other distributed storage systems, replication is important for document DBMSs in order to guarantee availability. The network bandwidth required to keep replicas synchronized is expensive and is often a performance bottleneck. As such, there is a strong need to reduce the replication bandwidth, especially for geo-replication scenarios where wide-area network (WAN) bandwidth is limited.

In this tutorial, we will teach how to use VM Hub (Visual Media Hub), an open multimedia hub with most of the code in the open source space, to convert a multimedia application to a cloud service, and to build mobile applications that consumes the cloud service. The tutorial also covers the architecture and design consideration of VM Hub.

In current peer-to-peer (P2P) live streaming systems, nodes in a channel form a P2P overlay for video sharing. To watch a new channel, a node depends on the centralized server to join in the overlay of the channel. In today's live streaming applications, the increase in the number of channels triggers users' desire of watching multiple channels successively or simultaneously. However, the support of such watching modes in current applications is no better than joining in different channel overlays successively or simultaneously through the centralized server, which if widely used, poses a heavy burden on the server. In order to achieve higher efficiency and scalability, we propose a Social-network-Aided efficient liVe strEaming system (SAVE). SAVE regards users' channel switching or multichannel watching as interactions between channels. By collecting the information of channel interactions, nodes' interests, and watching times, SAVE forms nodes in multiple channels with frequent interactions into an overlay, constructs bridges between overlays of channels with less frequent interactions, and enables nodes to identify friends sharing similar interests and watching times. Thus, a node can connect to a new channel while staying in its current overlay, using bridges or relying on its friends, reducing the need to contact the centralized server. We further propose the channel-closeness-based chunk-pushing strategy and capacity-based chunk provider selection strategy to enhance the system performance. Extensive experimental results from the PeerSim simulator and PlanetLab verify that SAVE outperforms other systems in system efficiency and server load reduction, as well as the effectiveness of the two proposed strategies.

Content Delivery Networks (CDNs) play a central role of today's Internet infrastructure, and have seen a sharp increment in scale. More and more internet sites are armed with dynamic (or live) content (such as live sports game statistics, e-commerce and online auction), and there is a need to deliver dynamic content freshly in scale. To achieve high scalability, the consistency maintenance problem for dynamic content (contents with frequent updates) served by CDNs is non-trivial. The large number of widely scattered replicas guarantee the service QoS of end-users, meanwhile largely increase the complexity of consistency maintenance. Current consistency maintenance infrastructures and methods cannot simultaneously satisfy the two requirements: scalability and consistency. In this paper, we first analyze our crawled trace data of a cached sports game content on thousands of content servers of a major CDN. We analyze the content consistency from different perspectives, from which we try to break down the reasons for inconsistency among content servers. Finally, we further evaluate the performance in consistency, scalability and overhead for different infrastructures with different update methods. We itemize the advantages and disadvantages of different methods and infrastructures in different scenarios through the evaluation. We aim to give guidance for appropriate selections of consistency maintenance infrastructures and methods for a CDN, and for choosing a CDN service with different considerations.

Recently, emerging research efforts have been focused on question and answer (Q&A) systems based on social networks. The social-based Q&A systems can answer non-factual questions, which cannot be easily resolved by web search engines. These systems either rely on a centralized server for identifying friends based on social information or broadcast a user's questions to all of its friends. Mobile Q&A systems, where mobile nodes access the Q&A systems through Internet, are very promising considering the rapid increase of mobile users and the convenience of practical use. However, such systems cannot directly use the previous centralized methods or broadcasting methods, which generate high cost of mobile Internet access, node overload, and high server bandwidth cost with the tremendous number of mobile users. We propose a distributed Social-based mObile Q&A System (SOS) with low overhead and system cost as well as quick response to question askers. SOS enables mobile users to forward questions to potential answerers in their friend lists in a decentralized manner for a number of hops before resorting to the server. It leverages lightweight knowledge engineering techniques to accurately identify friends who are able to and willing to answer questions, thus reducing the search and computation costs of mobile nodes. The trace-driven simulation results show that SOS can achieve a high query precision and recall rate, a short response latency and low overhead. We have also deployed a pilot version of SOS for use in a small group in Clemson University. The feedback from the users shows that SOS can provide high-quality answers.

Video sharing has been an increasingly popular application in online social networks (OSNs). However, its sustainable development is severely hindered by the intrinsic limit of the client/server architecture deployed in current OSN video systems, which is not only costly in terms of server bandwidth and storage but also not scalable with the soaring amount of users and video content. The peer-assisted Video-on-Demand (VoD) technique, in which participating peers assist the server in delivering video content, has been proposed recently. Unfortunately, videos can only be disseminated through friends in OSNs. Therefore, current VoD works that explore clustering nodes with similar interests or close location for high performance are suboptimal, if not entirely inapplicable, in OSNs. Based on our long-term real-world measurement of over 1,000,000 users and 2,500 videos on Facebook, we propose SocialTube, a novel peer-assisted video sharing system that explores social relationship, interest similarity, and physical location between peers in OSNs. Specifically, SocialTube incorporates four algorithms: a social network (SN)-based P2P overlay construction algorithm, an SN-based chunk prefetching algorithm, chunk delivery, and scheduling algorithm, and a buffer management algorithm. Experimental results from a prototype on PlanetLab and an event-driven simulator show that SocialTube can improve the quality of user experience and system scalability over current P2P VoD techniques.

The semantic gap between low-level visual features and high-level semantics has been investigated for decades but still remains a big challenge in multimedia. When "search" became one of the most frequently used applications, "intent gap", the gap between query expressions and users' search intents, emerged. Researchers have been focusing on three approaches to bridge the semantic and intent gaps: 1) developing more representative features, 2) exploiting better learning approaches or statistical models to represent the semantics, and 3) collecting more training data with better quality. However, it remains a challenge to close the gaps. In this paper, we argue that the massive amount of click data from commercial search engines provides a data set that is unique in the bridging of the semantic and intent gap. Search engines generate millions of click data (a.k.a. image-query pairs), which provide almost "unlimited" yet strong connections between semantics and images, as well as connections between users' intents and queries. To study the intrinsic properties of click data and to investigate how to effectively leverage this huge amount of data to bridge semantic and intent gap is a promising direction to advance multimedia research. In the past, the primary obstacle is that there is no such dataset available to the public research community. This changes as Microsoft has released a new large-scale real-world image click data to public. This paper presents preliminary studies on the power of large-scale click data with a variety of experiments, such as building large-scale concept detectors, tag processing, search, definitive tag detection, intent analysis, etc., with the goal to inspire deeper researches based on this dataset.

Real-time communication (RTC) applications such as VoIP, video conferencing, and online gaming are flourishing. To adapt and deliver good performance, these applications require accurate estimations of short-term network performance metrics, e.g., loss rate, one-way delay, and throughput. However, the wide variation in mobile cellular network performance makes running RTC applications on these networks problematic. To address this issue, various performance adaptation techniques have been proposed, but one common problem of such techniques is that they only adjust application behavior reactively after performance degradation is visible. Thus, proactive adaptation based on accurate short-term, fine-grained network performance prediction can be a preferred alternative that benefits RTC applications. In this study, we show that forecasting the short-term performance in cellular networks is possible in part due to the channel estimation scheme on the device and the radio resource scheduling algorithm at the base station. We develop a system interface called PROTEUS, which passively collects current network performance, such as throughput, loss, and one-way delay, and then uses regression trees to forecast future network performance. PROTEUS successfully predicts the occurrence of packet loss within a 0.5s time window for 98% of the time windows and the occurrence of long one-way delay for 97% of the time windows. We also demonstrate how PROTEUS can be integrated with RTC applications to significantly improve the perceptual quality. In particular, we increase the peak signal-to-noise ratio of a video conferencing application by up to 15dB and reduce the perceptual delay in a gaming application by up to 4s.

In this paper, we revisit the problem of multi-party conferencing from a practical perspective, and to rethink the design space involved in this problem. We believe that an emphasis on low end-to-end delays between any two parties in the conference is a must, and the source sending rate in a session should adapt to bandwidth availability and congestion. We present Celerity, a multi-party conferencing solution specifically designed to achieve our objectives. It is entirely Peer-to-Peer (P2P), and as such eliminating the cost of maintaining centrally administered servers. It is designed to deliver video with low end-to-end delays, at quality levels commensurate with available network resources over arbitrary network topologies where bottlenecks can be anywhere in the network. This is in contrast to commonly assumed P2P scenarios where bandwidth bottlenecks reside only at the edge of the network. The highlight in our design is a distributed and adaptive rate control protocol, that can discover and adapt to arbitrary topologies and network conditions quickly, converging to efficient link rate allocations allowed by the underlying network. In accordance with adaptive link rate control, source video encoding rates are also dynamically controlled to optimize video quality in arbitrary and unpredictable network conditions. We have implemented Celerity in a prototype system, and demonstrate its superior performance over existing solutions in a local experimental testbed and over the Internet.

We design flexible schemes to explore the tradeoffs between storage space and access efficiency in reliable data storage systems. Aiming at this goal, two new classes of erasure-resilient codes are introduced -- Basic Pyramid Codes (BPC) and Generalized Pyramid Codes (GPC). Both schemes require slightly more storage space than conventional schemes, but significantly improve the critical performance of read during failures and unavailability. As a by-product, we establish a necessary matching condition to characterize the limit of failure recovery, that is, unless the matching condition is satisfied, a failure case is impossible to recover. In addition, we define a maximally recoverable (MR) property. For all ERC schemes holding the MR property, the matching condition becomes sufficient, that is, all failure cases satisfying the matching condition are indeed recoverable. We show that GPC is the first class of non-MDS schemes holding the MR property.

Internet-based video streaming applications are becoming more and more popular, attracting millions of online viewers every day. The incredible growth of viewers, dynamics of participants, and high video quality of service (QoS) requirement pose scalability, availability, and low-latency challenges to peer-to-peer (P2P) live video streaming systems. Tree-based systems have low-delay but are vulnerable to churn, while mesh-based systems are churn-resilient but suffer from high delay and overhead. Also, both structures cannot make full utilization of the bandwidth in the system. To tackle the challenges, we propose a DHT-aided chunk-driven overlay (DCO). It introduces a scalable DHT ring structure into a mesh-based overlay to efficiently manage video stream sharing. DCO includes a two-layer hierarchical DHT-based infrastructure, a chunk sharing algorithm, and a video provider selection algorithm. It selects stable nodes to form a scalable DHT-based infrastructure. The nodes in the DHT serve as distributed matchmakers between video providers and requesters. To motivate stable nodes to serve as the DHT nodes, we introduce an incentive mechanism based on the game theory. Aided by DHT, DCO guarantees stream chunk availability and assigns to a chunk requester a provider among all available providers in the system so that stream chunks are transmitted along a dynamic tree with top-down decreasing node bandwidth. In this way, DCO takes full advantage of available bandwidth in the system and, at the same time, provides high scalability and low latency. Experimental results show the superior performance of DCO compared with mesh-based and tree-based systems, and the effectiveness of the incentive mechanism and provider selection algorithm.

Mobile broadband networks using 3G and 4G technologies (such as EV-DO, HSPA, WiMAX, LTE) are rapidly becoming one of the prominent means to access the Internet. Multimedia consumption - requiring low delay, high bandwidth, or a combination of both - is projected to become a large portion of bandwidth utilization in mobile broadband networks. In this paper, we study the fundamental problem of how packet loss and delay vary as a function of the transmission rate over these networks. With extensive real-world measurement studies, we analyze the performance of a number of rate control algorithms commonly used in media transmission. We show that the variable nature of congestion signals (loss and delay) on these networks leads to an ultimate failure of existing rate control strategies to deliver adequate performance for multimedia applications. In addition, we show how a rate control algorithm derived from the utility maximization framework - which uses queuing delay as the primary congestion signal - can be modified to solve the challenging issues we have observed. By using a variable threshold to define when the network is congested, our proposed solution is able to achieve a significant improvement over algorithms that use fixed definitions of congestion.

Video sharing has been an increasingly popular application in online social networks (OSNs). However, its sustainable development is severely hindered by the intrinsic limit of the client/server architecture deployed in current OSN video systems, which is not only costly in terms of server bandwidth and storage but also not scalable. The peer-assisted Video-on-Demand (VOD) technique, in which participating peers assist the server in delivering video content has been proposed recently. Unfortunately, videos can only be disseminated through friends in OSNs. Therefore, current VOD works that explore clustering nodes with similar interests or close location for high performance are suboptimal, if not entirely inapplicable, in OSNs. Based on our long-term real-world measurement of over 1,000,000 users and 2,500 videos in Facebook, we propose SocialTube, a novel peer-assisted video sharing system that explores social relationship, interest similarity, and physical location between peers in OSNs. Specifically, SocialTube incorporates three algorithms: a social network (SN)-based P2P overlay construction algorithm, a SN-based chunk prefetch algorithm, and a buffer management algorithm. The trace driven based simulation results show that SocialTube can improve the quality of user experience and system scalability over current P2P VOD techniques.

Network codes designed specifically for distributed storage systems have the potential to provide dramatically higher storage efficiency for the same availability. One main challenge in the design of such codes is the exact repair problem: if a node storing encoded information fails, in order to maintain the same level of reliability we need to create encoded information at a new node. One of the main open problems in this emerging area has been the design of simple coding schemes that allow exact and low cost repair of failed nodes and have high data rates. In particular, all prior known explicit constructions have data rates bounded by 1/2. In this paper we introduce the first family of distributed storage codes that have simple look-up repair and can achieve rates up to 2/3. Our constructions are very simple to implement and perform exact repair by simple XORing of packets. We experimentally evaluate the proposed codes in a realistic cloud storage simulator and show significant benefits in both performance and reliability compared to replication and standard Reed-Solomon codes.

In current peer-to-peer (P2P) live streaming systems, nodes in a channel form a P2P overlay for video sharing. To watch a new channel, a node depends on the centralized server to join in the overlay of the channel. The increase in the number of channels in today's live streaming applications triggers users' desire of watching multiple channels successively or simultaneously. However, the support of such watching modes in current applications is no better than joining in different channel overlays successively or simultaneously, which if widely used, poses heavy burden on the centralized server. In order to achieve higher efficiency and scalability, we propose a Social network-Aided efficient liVe strEaming system (SAVE). SAVE regards users' channel switching or multi-channel watching as interactions between channels. By collecting the information of channel interactions and nodes' interests and watching times, SAVE forms nodes in multiple channels with frequent interactions into an overlay, constructs bridges between overlays of channels with less frequent interactions, and enables nodes to identify friends sharing similar interests and watching times. Thus, a node can connect to a new channel while staying in its current overlay, using bridges or relying on its friends, reducing the need to contact the centralized server. Extensive experimental results from the PeerSim simulator and PlanetLab verify that SAVE outperforms other popular protocols in system efficiency and server load reduction.

In this paper, we study the problem of utility maximization in peer-to-peer (P2P) systems, in which aggregate application-specific utilities are maximized by running distributed algorithms on P2P nodes, which are constrained by their uplink capacities. For certain P2P topologies, we show that routing along a linear number of trees per source can achieve the largest rate region that can be possibly obtained by intrasession and intersession network coding. This observation allows us to develop a simple multitree formulation for the problem. For the resulting nonstrictly concave optimization problem, we develop a Primal-dual distributed algorithm and prove its global convergence using our proposed sufficient conditions. These conditions are general and add understanding to the convergence of primal-dual algorithms under nonstrictly concave settings. We implement the proposed distributed algorithm in a peer-assisted multiparty conferencing system by utilizing only end-to-end delay measurements between P2P nodes. We demonstrate its superior performance through actual experiments on a LAN testbed and the Internet.

Online forums have long since been the most popular platform for people to communicate and share ideas. Nowadays, with the boom of multimedia sharing, users tend to share more and more with their online peers within online communities such as forums. The server-client model of forums has been used since its creation in the mid-1990s. However, this model has begun to fall short in meeting the increasing need of bandwidth and storage resources as an increasing number of people share more and more multimedia content. In this work, we first investigate the unique properties of forums based on the data collected from the Disney discussion boards. According to these properties, we design a scheme to support P2P-based multimedia sharing in forums called Multimedia Board (MBoard). Extensive trace-driven simulation results utilizing real trace data show that MBoard can significantly reduce the load on the server while maintaining a high quality of service for the users.

The convergence of games and online social platforms is an exploding phenomena. The continued success of social games hinges critically on the ability to deliver smooth and highly-interactive experiences to end-users. However, it is extremely challenging to satisfy the stringent performance requirements of online social games.

The bloom filter is a probabilistic data structure that provides a compact representation of a set of elements. To keep false positive probabilities low, the size of the bloom filter must be dimensioned a priori to be linear in the maximum number of keys inserted, with the linearity constant ranging typically from one to few bytes. A bloom filter is most commonly used as an in memory data structure, hence its size is limited by the availability of RAM space on the machine. As datasets have grown over time to Internet scale, so have the RAM space requirements of bloom filters. If sufficient RAM space is not available, we advocate that flash memory may serve as a suitable medium for storing bloom filters, since it is about one-tenth the cost of RAM per GB while still providing access times orders of magnitude faster than hard disk. We present BLOOMFLASH, a bloom filter designed for flash memory based storage, that provides a new dimension of trade off with bloom filter access times to reduce RAM space usage (and hence system cost). The simple design of a single flat bloom filter on flash suffers from many performance bottlenecks, including in-place bit updates that are inefficient on flash and multiple reads and random writes spread out across many flash pages for a single lookup or insert operation. To mitigate these performance bottlenecks, BLOOMFLASH leverages two key design innovations: (i) buffering bit updates in RAM and applying them in bulk to flash that helps to reduce random writes to flash, and (ii) a hierarchical bloom filter design consisting of component bloom filters, stored one per flash page, that helps to localize reads and writes on flash. We use two real-world data traces taken from representative bloom filter applications to drive and evaluate our design. BLOOMFLASH achieves bloom filter access times in the range of few tens of microseconds, thus allowing up to order of tens of thousands operations per sec.

Cloud service providers operate data centers around the world, and they depend on Global Traffic Management systems to direct requests from clients to the most appropriate data center to serve the requests. While GTM systems have been in-use for years, they are attracting re-newed interests due to the rapid expansion of cloud service providers' networks, the introduction of public DNS systems, as well as new proposals to alter how they should work and what information local DNS servers (LDNS) should make available to drive the GTM systems. This paper uses large-scale measurements conducted from more than 5M clients to establish properties of the current Internet that affect the design of the GTM systems, such as the stretch between a client's actual position and its LDNS from GTM's perspective, the impact of public DNS systems, and the granularity at which GTM decisions should be made. The results can inform the debate over how GTM systems should be designed.

Online forums have long since been the most popular platform for people to communicate and share ideas. Nowadays, with the boom of multimedia sharing, users tend to share more and more with their online peers within online communities such as forums. The server-client model of forums has been used since its creation in the mid-nineties. However, this model has begun to fall short in meeting the increasing need of bandwidth and storage as an increasing number of people share more and more multimedia content. In this work, we first investigate the unique properties of forums based on the data collected from the Disney discussion boards. According to these properties, we design a scheme to support P2P-based multimedia sharing in forums called Multimedia Board (MBoard). Extensive simulation results utilizing real trace data show that MBoard can significantly reduce the load on the server while maintaining a high quality of service for the users.

To optimize network performance, cloud service providers have a number of options available to them, including co-locating production servers in well-connected Internet eXchange (IX) points, deploying data centers in additional locations, or contracting with external Content Distribution Networks (CDNs). Some of these options can be very costly, and some may or may not improve performance significantly. Cloud service providers would clearly like to be able to estimate a priori performance gain of the various options before sinking significant capital expenditures into major infrastructure changes. In this paper we take a measurement-oriented approach and develop methodologies that accurately predict the performance improvement for making major infrastructure changes. Our methodologies leverage active web content, existing large-scale CDN infrastructures, and the SpeedTest network. We then apply our methodologies and a CloudBeacon tool to the problem of locating satellite data centers throughout the world. The results show that for North America, a deployment limited to 11 locations will be sufficient. However, in order to provide good latency and throughput performance on a global scale, somewhere between a total of 36 and 72 cloud-service locations with good peering connections is most likely needed.

We consider exact repair of failed nodes in maximum distance separable (MDS) code based distributed storage systems. It is well known that an (n, k) MDS code can tolerate failure (erasure) of up to n - k storage disks, when the code is used to store k information elements over n distributed storage disks. The focus of this paper is optimal recovery, in terms of repair bandwidth - the amount of data to be downloaded to repair a failed node - for a single failed node. When a single node fails, it has been previously shown by Dimakis et. al. that the amount of repair bandwidth is at least equation units, when each storage disk stores ℒ units of data. The achievability of this lower bound of equation units, for arbitrary values of (n, k); has been shown previously using asymptotic code constructions based on asymptotic interference alignment. However, the existence of finite codes satisfying this lower bound has been shown only for specific regimes of (n, k) and their existence for arbitrary values of (n, k) remained open. In this paper, we provide the first known construction of a finite code for arbitrary (n, k), which can repair a single failed systematic node by downloading exactly equation units of data. The code that we construct is based on permutation matrices and hence termed the Permutation Code.

In this paper, we attempt to revisit the problem of multi-party conferencing from a practical perspective, and to rethink the design space involved in this problem. We believe that an emphasis on low end-to-end delays between any two parties in the conference is a must, and the source sending rate in a session should adapt to bandwidth availability and congestion. We present Celerity, a multi-party conferencing solution specifically designed to achieve our objectives. It is entirely Peer-to-Peer (P2P), and as such eliminating the cost of maintaining centrally administered servers. It is designed to deliver video with low end-to-end delays, at quality levels commensurate with available network resources over arbitrary network topologies where bottlenecks can be anywhere in the network. This is in contrast to commonly assumed P2P scenarios where bandwidth bottlenecks reside only at the edge of the network. The highlight in our design is a distributed and adaptive rate control protocol, that can discover and adapt to arbitrary topologies and network conditions quickly, converging to efficient link rate allocations allowed by the underlying network. In accordance with adaptive link rate control, source video encoding rates are also dynamically controlled to optimize video quality. We have implemented Celerity in a prototype system, and demonstrate its superior performance over existing solutions in a local experimental testbed and over the Internet.

In this paper, we attempt to revisit the problem of multi-party conferencing from a practical perspective, and to rethink the design space involved in this problem. We believe that an emphasis on low end-to-end delays between any two parties in the conference is a must, and the source sending rate in a session should adapt to bandwidth availability and congestion. We present Celerity, a multi-party conferencing solution specifically designed to achieve our objectives. It is entirely Peer-to-Peer (P2P), and as such eliminating the cost of maintaining centrally administered servers. It is designed to deliver video with low end-to-end delays, at quality levels commensurate with available network resources over arbitrary network topologies where bottlenecks can be anywhere in the network. This is in contrast to commonly assumed P2P scenarios where bandwidth bottlenecks reside only at the edge of the network. The highlight in our design is a distributed and adaptive rate control protocol, that can discover and adapt to arbitrary topologies and network conditions quickly, converging to efficient link rate allocations allowed by the underlying network. In accordance with adaptive link rate control, source video encoding rates are also dynamically controlled to present the best possible video quality in arbitrary and unpredictable network conditions. We have implemented Celerity in a prototype system and demonstrate its superior performance in a local experimental testbed.

We present SkimpyStash, a RAM space skimpy key-value store on flash-based storage, designed for high throughput, low latency server applications. The distinguishing feature of SkimpyStash is the design goal of extremely low RAM footprint at about 1 (± 0.5) byte per key-value pair, which is more aggressive than earlier designs. SkimpyStash uses a hash table directory in RAM to index key-value pairs stored in a log-structured manner on flash. To break the barrier of a flash pointer (say, 4 bytes) worth of RAM overhead per key, it "moves" most of the pointers that locate each key-value pair from RAM to flash itself. This is realized by (i) resolving hash table collisions using linear chaining, where multiple keys that resolve (collide) to the same hash table bucket are chained in a linked list, and (ii) storing the linked lists on flash itself with a pointer in each hash table bucket in RAM pointing to the beginning record of the chain on flash, hence incurring multiple flash reads per lookup. Two further techniques are used to improve performance: (iii) two-choice based load balancing to reduce wide variation in bucket sizes (hence, chain lengths and associated lookup times), and a bloom filter in each hash table directory slot in RAM to disambiguate the choice during lookup, and (iv) compaction procedure to pack bucket chain records contiguously onto flash pages so as to reduce flash reads during lookup. The average bucket size is the critical design parameter that serves as a powerful knob for making a continuum of tradeoffs between low RAM usage and low lookup latencies. Our evaluations on commodity server platforms with real-world data center applications show that SkimpyStash provides throughputs from few 10,000s to upwards of 100,000 get-set operations/sec.

Peer-to-peer (P2P) systems provide a scalable way to stream content to multiple receivers over the Internet. The maximum rate achievable by all receivers is the capacity of a P2P streaming session. We provide a taxonomy of sixteen problem formulations, depending on whether there is a single P2P session or there are multiple concurrent sessions, whether the given topology is a full mesh graph or an arbitrary graph, whether the number of peers a node can have is bounded or not, and whether there are nonreceiver relay nodes or not. In each formulation, computing P2P streaming capacity requires the computation of an optimal set of multicast trees, with an exponential complexity, except in three simplest formulations that have been recently solved with polynomial time algorithms. These solutions, however, do not extend to the other more general formulations. In this paper, we develop a family of constructive, polynomial-time algorithms that can compute P2P streaming capacity and the associated multicast trees, arbitrarily accurately for seven formulations, to a factor of 4-approximation for two formulations, and to a factor of log of the number of receivers for two formulations. The optimization problem is reformulated in each case so as to convert the combinatorial problem into a linear program with an exponential number of variables. The linear program is then solved using a primal-dual approach. The algorithms combine an outer loop of primal-dual update with an inner loop of smallest price tree construction, driven by the update of dual variables in the outer loop. We show that when the construction of smallest price tree can be carried out arbitrarily accurately in polynomial time, so can the computation of P2P streaming capacity. We also develop several efficient algorithms for smallest price tree construction. Using the developed algorithms, we investigate the impact of several factors on P2P streaming capacity using topologies derived from statistics of uplink capacities of Internet hosts.

As cloud computing is taking off, the presence of high-performance interactive Internet applications is exploding. By nature, these applications require responsive client-server data exchange and lossless, in-order delivery. Previous work has shown that by using forward error correction (FEC), it is possible to reduce the data streaming latency caused by retransmissions of lost packets. However, the prior schemes only send FEC packets when there are no original packets pending transmission. In this paper, we further expand the hybrid FEC-ARQ protocol and show that sometimes, the transmission latency can be further reduced by preempting original data packets with FEC packets. We have formulated the decision of whether to send new original data packets, FEC packets, or resend original data packets as a transmission policy. An optimal transmission policy is selected to minimize the delay experienced by the application subject to a constraint on the amount of overhead. By using this optimal policy, we significantly improve the delay performance over straightforward FEC schemes while controlling the amount of overhead due to FEC.

We consider multi-rate peer-to-peer multiparty video conferencing applications, where different receivers in the same group can receive videos at different rates using, for example, scalable layered coding. The quality of video received by each receiver can be modeled as a concave utility function of the video bitrate. We study and address the unique challenges introduced by maximizing utility in the multi-rate setting as compared to the single-rate case. We first determine an optimal set of tree structures for routing multi-rate content using scalable layered coding. We then develop Primal and Primal-dual based distributed algorithms to maximize aggregate utility of all receivers in all groups by multi-tree routing and show their convergence. These algorithms can be easily implemented and deployed on today's Internet. We have built a prototype video conferencing system to show that this approach converges to optimal bitrates to improve user experience and offers automatic adaptation to network conditions and user preferences.

There has been a dramatic increase in interactive cloud based software applications. Compared to classical real-time media applications (voice over IP (VoIP)/conferencing) and non real-time file delivery, these interactive software applications have unique characteristics: 1) they are delay sensitive yet demand in order and reliable data delivery, and 2) the traffic is usually bursty. Traditional window based congestion control does not work well for interactive applications because the bursty arrival of data leads to bursty network traffic, which causes additional queuing delay and packet loss in the network which reduces its delay performance. In this paper, we propose a new hybrid window plus rate based congestion control technique. This algorithm improves the delay performance of interactive applications by preventing congestion induced loss and minimizing queuing delay while still fully utilizing network capacity and maintaining fairness across multiple flows.

Voice over Internet (VoIP) has been experiencing enormous growth in recent years. While posed to replace traditional PSTN for both enterprise and residential customers, VoIP has yet to achieve the same level of quality and reliability as PSTN. One key challenge is that a growing segment of customers is increasingly relying on WiFi connections. VoIP over WiFi (VoWiFi) experiences significant degradation in quality because of packet losses, mostly due to WiFi's low capacity, varying signal strength, interference, etc. To understand this problem, we have developed and deployed a comprehensive measurement platform in a global enterprise network. From large-scale real-world traces, we quantitatively analyze the impact of WiFi connections and study measures to mitigate such impact. Our results confirm that WiFi connections incur significantly more packet losses than wirelines, but these losses can be effectively concealed by sending each packet up to five times (heavy replication). Due to WiFi's inherent overhead, heavy replication only marginally increases WiFi airtime. To avoid the overhead on wirelines, we further propose a relay-based solution, where heavy replication only occurs between endpoints and nearby relays, and is removed before packets are transmitted on inter-branch long haul links or the public Internet. The solution has been implemented and deployed in the global enterprise network, and measurement results confirm that it can indeed greatly improve the performance of VoIP for WiFi users. In particular, it reduces the percentage of poor calls from 35% to 10%; and increases the percentage of acceptable ones from 45% to 70%.

Two of the fundamental problems in peer-to-peer (P2P) streaming are as follows: what is the maximum streaming rate that can be sustained for all receivers, and what peering algorithms can achieve close to this maximum? These problems of computing and approaching the P2P streaming capacity are often challenging because of the constraints imposed on overlay topology. In this paper, we focus on the limit of P2P streaming rate under node degree bound, i.e., the number of connections a node can maintain is upper bounded. We first show that the streaming capacity problem under node degree bound is NP Complete in general. Then, for the case of node out-degree bound, through the construction of a “Bubble algorithm”, we show that the streaming capacity is at least half of that of a much less restrictive and previously studied case, where we bound the node degree in each streaming tree but not the degree across all trees. Then, for the case of node total-degree bound, we develop a “Cluster-Tree algorithm” that provides probabilistic guarantee of achieving a rate close to the maximum rate achieved under no degree bound constraint, when the node degree bound is logarithmic in network size. The effectiveness of these algorithms in approaching the capacity limit is demonstrated in simulations using uplink bandwidth statistics of Internet hosts. Both analysis and numerical experiments show that peering in a locally dense and globally sparse manner achieves near-optimal streaming rate if the degree bound is at least logarithmic in network size.

There is an ongoing explosion of interactive Internet applications. By nature, these applications require responsive client-server data exchange and lossless, in-order delivery. In previous work, we have shown that a hybrid FEC-ARQ protocol which combines sending original data packets with forward error correction (FEC) packets is effective in reducing the latency caused by retransmissions of lost packets. However, the prior scheme only sends FEC packets when there are no original packets pending transmission. In this paper, we further expand the investigation of the hybrid FEC-ARQ protocol and show that sometimes, the transmission latency can be further reduced by preempting original data packets with FEC packets. We have formulated the decision of whether to send new original data packets, FEC packets, or resend original data packets as a transmission policy. An optimal transmission policy is selected to minimize the delay experienced by the application subject to a constraint on the amount of waste. By using this optimal policy, we significantly improve the delay performance over straightforward FEC schemes while controlling the amount of waste due to FEC.

Internet-based video streaming applications is becoming more and more popular and attract millions of online viewers every day. The incredible growth of viewers, dynamics of participants, and high video quality-of-service (QoS) requirement pose scalability, availability and low-latency challenges to peer-to-peer (P2P) live video streaming systems. Tree-based systems have low-delay but are vulnerable to churn, while mesh-based systems are churn-resilient but suffers high delay and overhead. Both systems cannot make full utilization of the bandwidth in the system. To tackle the challenges, we propose a DHT-aided Chunk-driven Overlay (DCO). It introduces a scalable DHT ring structure into a mesh-based overlay to efficiently manage video stream sharing. DCO includes a hierarchical DHT-based infrastructure and a chunk sharing algorithm. Aided by DHT, DCO guarantees stream chunk availability. In this way, DCO flexibly takes full advantage of available bandwidth in the system and at the same time provides high scalability and low-latency. Simulation results show the superiority of DCO compared with mesh-based and tree-based systems.

Global enterprises are increasingly adopting unified communication solutions over traditional telephone systems. Such solutions provide integrated audio/video conferencing and messaging services, and enable flexible working environments by allowing mobile and dispersed users to communicate and collaborate easily and efficiently. The ultimate goal of unified communications is to ensure a smooth and best possible user experience across all scenarios. To address this challenge and understand the impact of various network scenarios on unified audio/video conferencing, we have developed a distributed experimental platform - SureCall - and deployed it on over 80 machines across a global enterprise and many residential networks. SureCall has collected worth of more than 6 months of packet-level audio/video conferencing traces. Through in-depth analysis of these traces, we have quantitatively compared how key performance metrics, such as packet loss and jitter, as well as the correlation between them, are affected by the enterprise and residential networks, by WiFi connections and VPN links, etc. In addition, we show how SureCall can serve as an ideal platform to design, experiment and validate new schemes and algorithms. We have developed a new audio quality classifier using the SureCall platform, which is being experimented with the recent release of Office Communicator solution for large-scale validation.

Recently, there has been a dramatic increase in interactive cloud based software applications (e.g. working on remote machines, online games, interactive websites such as financial, web search) and other soft real-time applications (traffic within data center). Compared to classical real-time media applications (VoIP/conferencing) and non real-time file delivery, these interactive software applications have unique characteristics as they are delay sensitive yet demand in order and reliable data delivery. Therefore existing protocols for delivery of lossless data (such as TCP) and other delivery protocols using UDP do not work well. In this demo, we show substantially improved performance for such traffic by using a transport protocol built on top of UDP which uses intelligent adaptive forward error correction (FEC) and improved congestion control (rate control). The transport protocol is made lossless by using a hybrid FEC/ARQ strategy. The congestion control technique improves the delay performance by preventing congestion induced loss and minimizing queuing delay while still fully utilizing network capacity and maintaining fairness across flows. In this demo, we present RAPID (a ReliAble transport Protocol for Improving end-to-end Delay) and show its effectiveness in improving the performance of interactive client-server applications.

We present FlashStore, a high throughput persistent key-value store, that uses flash memory as a non-volatile cache between RAM and hard disk. FlashStore is designed to store the working set of key-value pairs on flash and use one flash read per key lookup. As the working set changes over time, space is made for the current working set by destaging recently unused key-value pairs to hard disk and recycling pages in the flash store. FlashStore organizes key-value pairs in a log-structure on flash to exploit faster sequential write performance. It uses an in-memory hash table to index them, with hash collisions resolved by a variant of cuckoo hashing. The in-memory hash table stores compact key signatures instead of full keys so as to strike tradeoffs between RAM usage and false flash read operations. FlashStore can be used as a high throughput persistent key-value storage layer for a broad range of server class applications. We compare FlashStore with BerkeleyDB, an embedded key-value store application, running on hard disk and flash separately, so as to bring out the performance gain of FlashStore in not only using flash as a cache above hard disk but also in its use of flash aware algorithms. We use real-world data traces from two data center applications, namely, Xbox LIVE Primetime online multi-player game and inline storage deduplication, to drive and evaluate the design of FlashStore on traditional and low power server platforms. FlashStore outperforms BerkeleyDB by up to 60x on throughput (ops/sec), up to 50x on energy efficiency (ops/Joule), and up to 85x on cost efficiency (ops/sec/dollar) on the evaluated datasets.

Interactive Internet applications that rely on sequential streams for lossless data exchange often use retransmission protocols (e.g. TCP) for reliability and the guarantee of sequential data ordering. More so than for bulk file transfer or media delivery, lossless sequential streaming poses an even greater challenge for the common problem cases of retransmission protocols, such as lossy links or long network paths, manifesting as significant latency in the interactive user experience. We propose a hybrid FEC-ARQ protocol built on a packet streaming code that reduces to a simple strategy over sending or resending original data packets or check packets combining undecoded packets, based on actual network conditions. Experimental results show that our proposed protocol can significantly improve the total delay over retransmission and other schemes that use FEC, under a range of bandwidth and loss scenarios.

We consider multi-rate peer-to-peer multi-party conferencing applications, where different receivers in the same group can receive videos at different rates using, for example, scalable layered coding. The quality of video received by each receiver can be modeled as a concave utility function of the video rate. We study and address the unique challenges introduced by multi-rate setting as compared to the single-rate case. We first determine an optimal set of tree structures for routing multi-rate content using scalable layered coding. We then develop primal and primal-dual based distributed algorithms to maximize aggregate utility of all receivers in all groups by multi-tree routing and show their convergence. These algorithms can be easily implemented and deployed on today's Internet. We have built a prototype video conferencing system to show that this approach offers low end-to-end delay, low complexity and high throughput, along with automatic adaptation to network conditions and user preferences.

Motivated by P2P file transfer applications (e.g., BitTorrent) on the Internet, this paper considers the problem of delivering a file from a server to multiple receivers in a P2P network. Each receiver has an associated delay in receiving the file. We aim at understanding the optimal delay region, i.e., the set of all possible delay vectors that can be achieved. Previous work has addressed the problem of delivering the file to all receivers in minimum amount of time (equivalently, minimizing the maximum delay to the receivers), assuming peer uplinks are the only bottleneck in the network. This paper shows that it is in fact possible to significantly reduce the average delay at a slight increase in the maximum delay. Moreover, given an order at which the receivers finish downloading, the optimal delay region is characterized by a system of linear inequalities. Any point in the optimal delay region can be achieved by linear network coding. We also propose a simple routing scheme that has near-optimal empirical performance.

We address the following question in this study: Can a network application detect not only the occurrence, but also the location of congestion? Answering this question will not only help the diagnostic of network failure and monitor server's QoS, but also help developers to engineer transport protocols with more desirable congestion avoidance behavior. The paper answers this question through new analytic results on the two underlying technical difficulties: 1) synchronization effects of loss and delay in TCP, and 2) distributed hypothesis testing using only local loss and delay data. We present a practical Congestion Location Detection (CLD) algorithm that effectively allows an end host to distributively detect whether congestion happens in the local access link or in more remote links. We validate the effectiveness of CLD algorithm with extensive experiments.

We consider multi-source multicast communication scenarios in which each node has an aggregate outbound traffic capacity and can directly communicate with any other node. This is motivated by peer-to-peer (P2P) information dissemination applications on the Internet in which the uplink capacity of nodes is usually the bottleneck, being several times smaller than the downlink capacity. We also allow the communication in a group to be helped by non-receiver nodes (with respect to that group) as relays. Extending an earlier result for the single source case, we show that when coding is not allowed across sources, routing is optimal. Also, as a rather surprising discovery, we show that when all groups have pairwise identical or disjoint receivers, routing is optimal even when coding across sources is allowed. Moreover, routing along a linear number of trees per source is sufficient to achieve this. The latter scenario is common in multiparty conferencing systems, hence our results have interesting practical applications in the design of infrastructure-less P2P multiparty conferencing systems.

Peer-to-peer (P2P) content distribution is able to greatly reduce dependence on infrastructure servers and scale up to the demand of the Internet video era. However, the rapid growth of P2P applications has also created immense burden on service providers by generating significant ISP-unfriendly traffic, such as cross-ISP and inter-POP traffic. In this work, we consider the unique properties of peer-assisted Video-on-Demand (VoD) and design a distributed rate allocation algorithm, which can significantly cut down on ISP-unfriendly traffic without much impact on server load. Through extensive packet-level simulation with both synthetic and real-world traces, we show that the rate allocation algorithm can achieve substantial additional gain, on top of previously proposed schemes advocating ISP-friendly topologies.

In this paper, we quantify the potential gains of hybrid CDN-P2P for two of the leading CDN companies, Akamai and Limelight. We first develop novel measurement methodology for mapping the topologies of CDN networks. We then consider ISP-friendly P2P distribution schemes which work in conjunction with the CDNs to localize traffic within regions of ISPs. To evaluate these schemes, we use two recent, real-world traces: a video-on-demand trace and a large-scale software update trace. We find that hybrid CDN-P2P can significantly reduce the cost of content distribution, even when peer sharing is localized within ISPs and further localized within regions of ISPs. We conclude that hybrid CDN-P2P distribution can economically satisfy the exponential growth of Internet video content without placing an unacceptable burden on regional ISPs.

In this paper, we study the problem of utility maximization in P2P systems, in which aggregate application-specific utilities are maximized by running distributed algorithms on P2P nodes, which are constrained by their uplink capacities. This may be understood as extending Kelly's seminal framework from single-path unicast over general topology to multi-path multicast over P2P topology, with network coding allowed. For certain classes of popular P2P topologies, we show that routing along a linear number of trees per source can achieve the largest rate region that can be possibly obtained by (multi-source) network coding. This simplification result allows us to develop a new multi-tree routing formulation for the problem. Despite of the negative results in literature on applying Primal-dual algorithms to maximize utility under multi-path settings, we have been able to develop a Primal-dual distributed algorithm to maximize the aggregate utility under the multi-path routing environments. Utilizing our proposed sufficient condition, we show global exponential convergence of the Primal-dual algorithm to the optimal solution under different P2P communication scenarios we study. The algorithm can be implemented by utilizing only end-to-end delay measurements between P2P nodes; hence, it can be readily deployed on today's Internet. To support this claim, we have implemented the Primal-dual algorithm for use in a peer-assisted multi-party conferencing system and evaluated its performance through actual experiments on a LAN testbed and the Internet.

In this paper, we study the maximally recoverable (MR) property for multi-protection group (MPG) codes. MPG codes with MR property achieve the best erasure recoverability given configurations, where a configuration represents the structural relationship between data and parity symbols. We present construction and decoding algorithms for MPG codes with MR property. We show that both recoverability and minimum decoding overhead of any MPG code with MR property depend only on the configuration, where decoding overhead is defined as the additional number of symbols to access, in order to decode the lost data symbols.

To flexibly explore the trade-offs between storage space and access efficiency in reliable data storage systems, we describe two classes of erasure resilient coding schemes: basic and generalized pyramid codes. The basic pyramid codes can be simply derived from any existing codes, and thus all known efficient encoding/decoding techniques directly apply. The generalized pyramid codes are radically advanced new codes, which can further improve access efficiency and/or reliability upon the basic pyramid codes. We also establish a necessary condition for any failure pattern to be ever recoverable, and show that the generalized pyramid codes are optimal in failure recovery (i.e., the necessary condition is also sufficient, and any failure pattern that is ever recoverable can indeed be recovered).

Video-on-demand in the Internet has become an immensely popular service in recent years. But due to its high bandwidth requirements and popularity, it is also a costly service to provide. We consider the design and potential benefits of peer-assisted video-on-demand, in which participating peers assist the server in delivering VoD content. The assistance is done in such a way that it provides the same user quality experience as pure client-server distribution. We focus on the single-video approach, whereby a peer only redistributes a video that it is currently watching.

The eleven papers in this special issue are devoted to advancements in peer-to-peer streaming systems.

In this paper, we investigate the use of erasure resilient code (ERC) in a peer-to-peer storage cloud. We compare the random linear code (RLC, network coding) and the Reed-Solomon (RS) code, and study the proper ERC parameters in term of data reliability, computational efficiency and security concerns. We conclude that the use of ERC in P2P storage may greatly improve data reliability and reduce backup server cost. Because of the lack of efficient homomorphic hashing, RLC cannot be implemented efficiently and yet be effective against the malicious attack. As a result, we believe that the RS code is most suited for use in the P2P storage cloud

We introduce context modeling into the distributed source coding (DSC). By forming contexts from prior coded bitplanes of both the reference and DSC coded peer, we split the virtual channel between the two correlated bitplanes into several virtual sub-channels of different characteristics. The selection of the subchannel becomes side information that is known to the receiver. As a result, the DSC coding bitrate is reduced. We also investigate a number of practical implementation issues in DSC; e.g., the use of turbo-based channel code vs. the LDPC-based channel code, and the use of a random flipper to handle the binary asymmetric channel. We implemented a practical DSC audio coding system. We show that DSC without context modeling shows little performance gain compared with separate source coding with context modeling. In comparison, context DSC achieves an overall rate saving of 36%, even as we only applied DSC on a selected number of bitplanes, and considering the channel code loss in practical implementation.

The paper describes DISCOVR, a distributed collaborative video recorder. DISCOVR is a P2P application that combines asynchronous file sharing with synchronous on-demand media streaming. DISCOVR uses a flat entity ID space, with the entity being any of the media file, header, mega packets, index and metadata. All DISCOVR entities may be asynchronously or synchronously distributed. DISCOVR adopts a sender-driven priority based sharing protocol. If the user is on-demand viewing a media file, those packets that are to be viewed in the near future will be put on the synchronous access list, which prompts its connected peers and the peers that are indirectly connected to fulfill the distribution of the on-demand packets in high priority. By letting the peers engage in both asynchronous sharing and synchronous on-demand streaming, DISCOVR promotes the peers to remain online longer, thus improve the availability of the P2P system and the overall performance

We have developed PeerStreaming, an on-demand peer-to-peer (P2P) media streaming solution based on a receiver-driven streaming protocol. Unlike existing client-server based media streaming solution, where all clients go to a single server to retrieve the streaming media, in the PeerStreaming system, the media may be streamed from multiple peers. We use a receiver-driven protocol so that the peers need no collaboration (thus no network traffic) among themselves. Only the receiver (the media playing client) coordinates the streaming among the peers, performs load balancing, handles the online/offline of the peers, and decodes and renders the media

A reversible transform not only converts integer input to integer output, but also reconstructs the exact input from the output. It is one of the key modules for lossless and progressive-to-lossless media codecs. We have established that the closer the reversible transform is to its float counterpart, the better the compression performance of the lossless and progressive-to-lossless codecs that utilize the transform. In this work, a new design of the reversible transform based on matrix lifting is proposed. With matrix lifting, we can design a transform that is reversible, is close in transform value to its float counterpart, and can be computed via fast algorithms. A progressive-to-lossless embedded audio codec employing the reversible MDCT with matrix lifting is implemented. Superior results on lossless and lossy audio compression are demonstrated.

The concentric mosaic offers a solution for quick construction and easy navigation of a virtual environment. However, the huge amount of data presents a heavy burden. We propose the rebinning of slits (ROSS) for the compression of the concentric mosaic. The proposed scheme decorrelates the three-dimensional (3-D) concentric mosaic by rebinning the vertical slits of the 3-D image array into a large two-dimensional (2-D) panorama, and then it compresses the panorama with a JPEG 2000 image coder. Unlike typical video coders that use motion compensation to decorrelate multiple frames, the rebinning converts 3-D spatial correlation into stronger 2-D correlation. Experimental results show that the ROSS outperforms MPEG-2 coding of the concentric mosaic by an average of 1.0 dB.

Using an array of photo shots, the concentric mosaic offers a quick way to capture and model a realistic three-dimensional (3-D) environment. We compress the concentric mosaic image array with a 3-D wavelet transform and coding scheme. Our compression algorithm and bitstream syntax are designed to ensure that a local view rendering of the environment requires only a partial bitstream, thereby eliminating the need to decompress the entire compressed bitstream before rendering. By exploiting the ladder-like structure of the wavelet lifting scheme, the progressive inverse wavelet synthesis (PIWS) algorithm is proposed to maximally reduce the computational cost of selective data accesses on such wavelet compressed datasets. Experimental results show that the 3-D wavelet coder achieves high-compression performance. With the PIWS algorithm, a 3-D environment can be rendered in real time from a compressed dataset.

The concentric mosaic offers a quick solution to construct a virtual copy of a real environment, and to navigate in that virtual environment. However, the huge amount of data associated with the concentric mosaic is a heavy burden for its application. A three-dimensional (3D) wavelet-based compressor has been proposed in a previous work to compress the concentric mosaic. In this paper, we greatly improve the compression efficiency of the 3D wavelet coder by a data rearrangement mechanism called "smart rebinning". The proposed scheme first aligns the concentric mosaic image shots along the horizontal direction and then rebins the shots into multiperspective panoramas. Smart rebinning effectively enhances the correlation in the 3D data volume, translating the data into a representation that is more suitable for 3D wavelet compression. Experimental results show that the performance of the 3D wavelet coder is improved by an average of 4.3 dB with the use of the smart rebinning. The proposed coder outperforms MPEG-2 coding of the concentric mosaic by an average of 3.7 dB.

A motion compensated lifting (MCLIFT) framework is proposed for the 3D wavelet video coder. By using bi-directional motion compensation in each lifting step of the temporal direction, the video frames are effectively de-correlated. With proper entropy coding and bitstream packaging schemes, the MCLIFT wavelet video coder can be scalable in frame rate and quality level. Experimental results show that the MCLIFT video coder outperforms the 3D wavelet video coder with the same entropy coding scheme by an average of 1.1-1.6dB, and outperforms MPEG-4 coder by an average of 0.9-1.4dB.

Video-object coding is one of the most important functionalities proposed by MPEG-4. We propose a new wavelet method to encode the texture of an arbitrarily shaped object, both for the still and for the video-object. The method uses the shape adaptive wavelet transform (SA-DWT) in MPEG-4 still object coding, but with a computationally more efficient lifting implementation. The transformed object coefficients are then quantized and entropy encoded with a partial bit-plane embedded coder, which greatly improves the coding efficiency. We denote the coding algorithm as the video-object wavelet (VOW) coder. Experimental results show that VOW significantly outperforms MPEG-4 in still-object coding, and achieves a comparable performance in video-object coding in terms of PSNR. Moreover, the VOW decoded object looks better subjectively, with less annoying blocking artifacts than that of MPEG-4.

In image based rendering (IBR), a 3D scene is recorded through a set of photos, and a novel view is rendered by assembling data from the photo set. Compression is essential to reduce the huge data amount of IBR. We examine three categories of IBR compression algorithms: the block coder, the reference coder and the high dimensional transform (wavelet) coder. It is observed that the block coder consumes the least computation resource, however, its compression ratio is low. The reference coder achieves a good compression ratio with reasonable computation complexity. The high dimensional wavelet coder achieves the best compression ratio, however, it is also the most complex.

Video object coding is one of the most important functionalities proposed by MPEG4. In this paper, we propose a new wavelet method to encode the texture of an arbitrarily shaped object, both for the still and video object. The method uses the shape adaptive wavelet transform (SA-DWT) in MPEG4 still object coding, but with a computationally more efficient lifting implementation. The transformed object coefficients are then quantized and entropy encoded with a partial bitplane embedded coder, which greatly improves the coding efficiency. We denote the coding algorithm as a video object wavelet (VOW) coder. Experimental results show that VOW significantly outperforms MPEG4 in still object coding, and achieves a comparable performance in video object coding in terms of PSNR. Moreover, the VOW decoded object looks better subjectively, with less annoying blocking artifacts than that of MPEG4.

Concentric mosaics offer a quick solution to construct a virtual copy of a real environment, and navigate in the virtual environment. However, the huge amount of data associated with concentric mosaics is a heavy burden for its application. A 3D wavelet transform-based compressor has been proposed in previous work to compress the concentric mosaics. In this paper, we greatly improve the performance of the 3D wavelet coder with a data rearrangement mechanism called “smart rebinning”. The proposed scheme first aligns the concentric mosaic image shots along the horizontal direction and then rebins the shots into multi-perspective panoramas. Smart rebinning greatly improves the cross shot correlation and enables the coder to better explore the redundancy among shots. Experimental results show that the performance of the 3D wavelet coder improves an average of 4.3dB with the use of smart rebinning. The proposed coder outperforms MPEG-2 coding of concentric mosaics by an average of 3.7dB.

A hybrid wavelet-fractal coder (WFC) for image compression is proposed. The WFC uses the fractal contractive mapping to predict the wavelet coefficients of the higher resolution from those of the lower resolution and then encode the prediction residue with a bitplane wavelet coder. The fractal prediction is adaptively applied only to regions where the rate saving offered by fractal prediction justifies its overhead. A rate-distortion criterion is derived to evaluate the fractal rate saving and used to select the optimal fractal parameter set for WFC. The superior performance of the WFC is demonstrated with extensive experimental results.

A multiscale post processing algorithm based on constrained optimization with the Huber Markov random field (HMRF) model is investigated. The decoded image is enhanced from coarse to fine scales, where postprocessing at the coarse scale improves the global appearance of the image and reduces long range artifacts such as ringing while postprocessing at the fine scale keeps the sharpness of edges. The efficiency of the proposed algorithm is supported by experimental results.

Embedded coders provide a better rate-distortion tradeoff while the coded bit stream can be truncated at any point without a significant perceptible distortion. We investigate rate control for an embedded wavelet video coder by converting the rate control problem to a bit allocation problem for each frame. Then, a computationally efficient rate control algorithm is derived by exploiting the rate-distortion performance of the embedded wavelet coder and the frame dependency between the reference frame and the predictive frame. Experiments are performed to demonstrate the superior performance of the embedded wavelet video coder with the proposed rate control scheme. It is shown that the proposed rate control strategy outperforms the fixed allocation rate control by 0.1-0.4 dB for a variety of sequences, and the performance gain can be as large as 1.3-2.7 dB around scene changes.

Motivated by Shapiro's (1993) embedded zerotree wavelet (EZW) coding and Taubman and Zakhor's (1994) layered zero coding (LZC), we propose a layered discrete cosine transform (DCT) image compression scheme, which generates an embedded bit stream for DCT coefficients according to their importance. The new method allows progressive image transmission and simplifies the rate-control problem. In addition to these functionalities, it provides a substantial rate-distortion improvement over the JPEG standard when the bit rates become low. For example, we observe a bit rate reduction with respect to the JPEG Huffman and arithmetic coders by about 60% and 20%, respectively, for a bit rate around 0.1 b/p.

We propose a hybrid fractal wavelet coder (FWC) which adopts adaptive fractal prediction in the wavelet domain and wavelet coding in this research. This new compression scheme uses wavelet coding as the core component, and applies contractive mapping (or the so-called fractal prediction) whenever it offers additional rate savings. A model based rate-distortion (R-D) criterion of low complexity is used to evaluate the fractal prediction efficiency and search for the best domain-range match. The superior R-D performance of FWC is demonstrated with extensive experimental results.

Motivated by Shapiro's (see IEEE Trans. on Signal Processing, vol.41, no.12, p. 3445-62, 1993) embedded zerotree wavelet coding (EZW) and Taubman and Zakhor's (see IEEE Trans. on Image Processing, vol.3, no.3, p.572-88, 1994) layered zero coding (LZC), we propose a layered DCT image compression scheme, which generates an embedded bit stream for DCT coefficients according to their importance. The new method allows progressive image transmission and simplifies the rate-control problem. In addition to these functionalities, it provides a substantial rate-distortion improvement over the JPEG standard when the bit rates become low. For example, we observe a bit rate reduction with respect to the JPEG Huffman and arithmetic coders by about 60% and 20%, respectively, for a bit rate around 0.1 bpp.

A new framework for multiscale video coding is proposed in this research with the objective to achieve efficient video coding with spatial scalability. In this framework, multiscale motion compensation is performed to remove temporal redundancy, and both the motion field and the displacement frame difference (DFD) are compressed via wavelet transform, successive embedded quantization and entropy coding for multiscale representation and decoding. The proposed new method does not only secure the prediction loop of lower resolution video but also provides an efficient prediction from coarser to finer resolutions.

We examine an approach to perform pattern recognition on compressed video such as MPEG data streams. The current effort has focused on the identification of the region of interest (ROI) which contains the target, which is also known as the focus of attention (FOA) module. Two techniques are studied, i.e. motion field analysis and DCT transform coefficient analysis. The performance of the proposed algorithms is demonstrated with experimental results.

Motion estimation and compensation are important tasks in image sequence coding. In this paper, we present a motion estimation scheme with multiresolution tree structure and hierarchical motion vector search. Experiments and analysis show that this scheme is not only computation efficient, but also robust. The multiresolution tree structure is further utilized in a variable block size image sequence coding scheme that incorporates the visual spatio-temporal characteristics. Both DCT and quadtree approaches are used to encode the motion compensated prediction error. Although the signal-to-noise ratio of the quadtree coded image is a little lower, the subjective quality around sharp edges is much better. Comparing our extensive simulation results with the MPEG-1 standard, we have obtained quite promising results.