A substantial amount of work has recently gone into localizing BitTorrent traffic within an ISP in order to avoid excessive and often times unnecessary transit costs. Several architectures and systems have been proposed and the initial results from specific ISPs and a few torrents have been encouraging. In this work we attempt to deepen and scale our understanding of locality and its potential. Looking at specific ISPs, we consider tens of thousands of concurrent torrents, and thus capture ISP-wide implications that cannot be appreciated by looking at only a handful of torrents. Second, we go beyond individual case studies and present results for few thousands ISPs represented in our data set of up to 40K torrents involving more than 3.9M concurrent peers and more than 20M in the course of a day spread in 11K ASes. Finally, we develop scalable methodologies that allow us to process this huge data set and derive accurate traffic matrices of torrents. Using the previous methods we obtain the following main findings: i) Although there are a large number of very small ISPs without enough resources for localizing traffic, by analyzing the 100 largest ISPs we show that Locality policies are expected to significantly reduce the transit traffic with respect to the default random overlay construction method in these ISPs; ii) contrary to the popular belief, increasing the access speed of the clients of an ISP does not necessarily help to localize more traffic; iii) by studying several real ISPs, we have shown that soft speed-aware locality policies guarantee win-win situations for ISPs and end users. Furthermore, the maximum transit traffic savings that an ISP can achieve without limiting the number of inter-ISP overlay links is bounded by “unlocalizable” torrents with few local clients. The application of restrictions in the number of inter-ISP links leads to a higher transit traffic reduction but the QoS of clients downloading “unlocalizable” torrents would be severely harmed.

The difficulty of partitioning social graphs has introduced new system design challenges for scaling of online social networks (OSNs). Vertical scaling by resorting to full replication can be a costly proposition. Scaling horizontally by partitioning and distributing data among multiple servers using, for e.g., distributed hash tables (DHTs), can suffer from expensive interserver communication. Such challenges have often caused costly rearchitecting efforts for popular OSNs like Twitter and Facebook. We design, implement, and evaluate SPAR, a Social Partitioning and Replication middleware that mediates transparently between the application and the database layer of an OSN. SPAR leverages the underlying social graph structure in order to minimize the required replication overhead for ensuring that users have their neighbors' data colocated in the same machine. The gains from this are multifold: Application developers can assume local semantics, i.e., develop as they would for a single machine; scalability is achieved by adding commodity machines with low memory and network I/O requirements; and N+K redundancy is achieved at a fraction of the cost. We provide a complete system design, extensive evaluation based on datasets from Twitter, Orkut, and Facebook, and a working implementation. We show that SPAR incurs minimum overhead, can help a well-known Twitter clone reach Twitter's scale without changing a line of its application logic, and achieves higher throughput than Cassandra, a popular key-value store database.

Most known frequent item set mining algorithms work by enumerating candidate item sets and pruning infrequent candidates. An alternative method, which works by intersecting transactions, is much less researched. To the best of our knowledge, there are only two basic algorithms: a cumulative scheme, which is based on a repository with which new transactions are intersected, and the Carpenter algorithm, which enumerates and intersects candidate transaction sets. These approaches yield the set of so-called closed frequent item sets, since any such item set can be represented as the intersection of some subset of the given transactions. In this paper we describe a considerably improved implementation scheme of the cumulative approach, which relies on a prefix tree representation of the already found intersections. In addition, we present an improved way of implementing the Carpenter algorithm. We demonstrate that on specific data sets, which occur particularly often in the area of gene expression analysis, our implementations significantly outperform enumeration approaches to frequent item set mining.

A substantial amount of work has recently gone into localizing BitTorrent traffic within an ISP in order to avoid excessive and often times unnecessary transit costs. Several architectures and systems have been proposed and the initial results from specific ISPs and a few torrents have been encouraging. In this work we attempt to deepen and scale our understanding of locality and its potential. Looking at specific ISPs, we consider tens of thousands of concurrent torrents, and thus capture ISP-wide implications that cannot be appreciated by looking at only a handful of torrents. Secondly, we go beyond individual case studies and present results for the top 100 ISPs in terms of number of users represented in our dataset of up to 40K torrents involving more than 3.9M concurrent peers and more than 20M in the course of a day spread in 11K ASes. We develop scalable methodologies that allow us to process this huge dataset and get concrete quantitative answers rather than qualitative speculations to questions like: “what is the minimum and the maximum transit traffic reduction across hundreds of ISPs?”, “what are the win-win boundaries for ISPs and their users?”, “what is the maximum amount of transit traffic that can be localized without requiring fine-grained control of inter-AS overlay connections?”.

Large datacenter operators with sites at multiple locations dimension their key resources according to the peak demand of the geographic area that each site covers. The demand of specific areas follows strong diurnal patterns with high peak to valley ratios that result in poor average utilization across a day. In this paper, we show how to rescue unutilized bandwidth across multiple datacenters and backbone networks and use it for non-real-time applications, such as backups, propagation of bulky updates, and migration of data. Achieving the above is non-trivial since leftover bandwidth appears at different times, for different durations, and at different places in the world.

The difficulty of scaling Online Social Networks (OSNs) has introduced new system design challenges that has often caused costly re-architecting for services like Twitter and Facebook. The complexity of interconnection of users in social networks has introduced new scalability challenges. Conventional vertical scaling by resorting to full replication can be a costly proposition. Horizontal scaling by partitioning and distributing data among multiples servers - e.g. using DHTs - can lead to costly inter-server communication.

A substantial amount of work has recently gone into localizing BitTorrent traffic within an ISP in order to avoid excessive and often times unnecessary transit costs. In this work we aim to answer yet unanswered questions such as: what is the minimum and the maximum transit traffic reduction across hundreds of ISPs?, what are the win-win boundaries for ISPs and their users?, what is the maximum amount of transit traffic that can be localized without requiring fine-grained control of inter-AS overlay connections?, what is the impact to transit traffic from upgrades of residential broadband speeds?.

Although a P2P scheme based unicast forward or application level multicast (ALM) is able to decentralize the video delivery process to reduce the VoD server load, the performance of such a scheme is limited by the high network resource requirement of the clients' communications. In this paper, we propose a new P2P architecture that uses multicast communications for peer collaborations. We introduce the concept of n-peers to m-peers cooperative collaboration to achieve high client delivery efficiency as well as unbounded scalable client collaboration capacity. Furthermore, the architecture selects peers by taking into account the underlying network infrastructure in order to avoid network saturation. We compared the new design with video delivery architecture based on Patching and Chaining schemes in a synthetic network model as well as a real Ethernet network topology. The experimental results showed that our design is better than Patching and Chaining in terms of admitted client requests and VoD server resource requirement. Furthermore, the results showed that our architecture is adaptable to any underlying physical networks.

For a high scalable VoD system, the distributed server architecture (DVoD) with more than one server-node is a cost-effective design solution. However, such a design is highly vulnerable to workload variations because the service capacity is limited. In this paper, we propose a new and efficient VoD architecture that combines DVoD with a P2P system. The DVoD's server-nodes is able to offer a minimum required quality of service (QoS) and the P2P system is able to provide the mechanism to increase the system service capacity according to client demands. Our P2P system is able to synchronize a group of clients in order to replace server-nodes in the delivery process. We compared the new VoD architecture with DVoD architecture based on classic multicast and P2P delivery policies (patching and chaining). The experimental results showed that our design is better than previous solutions, providing higher scalability

Distributed video-on-demand servers (DVS) are proposed as a solution to the limited streaming capacity and null scalability of large-scale centralized systems. Server interconnection topology plays an important role in video-on-demand systems' performance. This paper presents an analysis of different topologies and their influence over storage management and distribution, delivery policies performance, refusing requests occurrence, network consumption and scalability. To accomplish the proposal study, we have designed a complete simulation framework for DVS systems. Experimental results obtained under different workload conditions allow us to draw two important conclusions: First, a better connectivity implies a lower mean request service distance and lesser network requirements, improving multicast policies efficiency. Second, topology regularity is essential, as it allows a greater traffic balancing and provides more alternative routing paths. The analysis of global results shows that hypercube presents the best trade-off among all the evaluated metrics, providing a gradual and unlimited scalability for the DVS system.

A VOD server has to employ admission control algorithms to decide whether a new client (request) can be admitted without violating the requirements of clients who are already being serviced. This paper presents and analyzes four specific admission control algorithms pertaining to the deterministic and predictive category. These algorithms are: maximum, adaptive maximum, average and adaptive average. They are studied in detail for CPU and network resources. The adaptive algorithms are our own contribution and we show that they use an exponential weighted average to guess the future resource needs of a stream, rather than just using the maximum or mean over the whole period. Thus, they provide smoothing and so a closer matching of resources used to those reserved and so higher capacity and less waiting time.