The demand for stream processing at Facebook has grown as services increasingly rely on real-time signals to speed up decisions and actions. Emerging real-time applications require strict Service Level Objectives (SLOs) with low downtime and processing lag-even in the presence of failures and load variability. Addressing this challenge at Facebook scale led to the development of Turbine, a management platform designed to bridge the gap between the capabilities of the existing general-purpose cluster management frameworks and Facebook's stream processing requirements. Specifically, Turbine features a fast and scalable task scheduler; an efficient predictive auto scaler; and an application update mechanism that provides fault-tolerance, atomicity, consistency, isolation and durability. Turbine has been in production for over three years, and one of the core technologies that enabled a booming growth of stream processing at Facebook. It is currently deployed on clusters spanning tens of thousands of machines, managing several thousands of streaming pipelines processing terabytes of data per second in real time. Our production experience has validated Turbine's effectiveness: its task scheduler evenly balances workload fluctuation across clusters; its auto scaler effectively and predictively handles unplanned load spikes; and the application update mechanism consistently and efficiently completes high scale updates within minutes. This paper describes the Turbine architecture, discusses the design choices behind it, and shares several case studies demonstrating Turbine capabilities in production.

Stream processing applications reduce the latency of batch data pipelines and enable engineers to quickly identify production issues. Many times, a service can log data to distinct streams, even if they relate to the same real-world event (e.g., a search on Facebook's search bar). Furthermore, the logging of related events can appear on the server side with different delay, causing one stream to be significantly behind the other in terms of logged event times for a given log entry. To be able to stitch this information together with low latency , we need to be able to join two different streams where each stream may have its own characteristics regarding the degree in which its data is out-of-order . Doing so in a streaming fashion is challenging as a join operator consumes lots of memory, especially with significant data volumes. This paper describes an end-to-end streaming join service that addresses the challenges above through a streaming join operator that uses an adaptive stream synchronization algorithm that is able to handle the different distributions we observe in real-world streams regarding their event times. This synchronization scheme paces the parsing of new data and reduces overall operator memory footprint while still providing high accuracy. We have integrated this into a streaming SQL system and have successfully reduced the latency of several batch pipelines using this approach.

Spelling errors when typing a URL can be exploited by website-squatters: users are led to phony sites in a phenomenon we call parasitic URL naming. These phony sites imitate popular websites and try to extract personal information from unsuspecting users, or simply advertise and sell products to users. In this paper, we conduct a massive study in order to quantify the extent of this parasitic URL naming We start with a corpus of 900 popular websites, which we refer to as original URLs, and generate roughly 3 million URLs by varying the original names systematically and exhaustively. Over a period of 60 days, we analyze how many sites have URLs very similar to our original URLs. We find that parasitic URL naming is a wide-spread problem and quantify the extent of this issue. We believe that this work will provide the first step towards research and tools to combat web-fraud.

Peer-to-Peer (P2P) networks have proven to be an efficient and successful mechanism for file sharing over the Internet. However, current P2P protocols have long worst case query latencies which prevents them from be employed for real time applications. Popularity of objects in these networks can change rapidly and augurs the need for a rapid and lightweight content replication strategy to reduce search and data-access latencies. In this paper, we propose an on-line pointer replication (OPR) algorithm in structured P2P networks which yields a significantly low worst case query latency. Also, the degree of replication achieved by OPR is dynamically adaptable to the instantaneous query arrival rate and churn characteristics of the system in order to reduce total control traffic. We evaluate and compare different replica placement strategies on the PlanetLab network as well as with simulations. Experimental results show that OPR outperforms the existing replica placement algorithms by at least 30% in average latency and around 40% in terms of maximum query latency.

YouTube like media portals have changed the way users access media content in the Internet. Among many popular media sharing websites, some of the most popular ones are YouTube [7], Google-Video [4], Yahoo-Video [6], Metacafe [5], Flickr [3] and BlipTV [1]. Two factors along with the ubiquity of the Web have bolstered tremendous growth of these sites, namely, availability and media quality.

Peer to peer (P2P) flows constitute a large portion of Internet traffic meandering through different ISP domains. Hence, it is of prime concern for ISPs to try and gauge the number of active P2P users and where they are located, both inside and outside their respective domains. An analysis of where P2P users are located in the Internet provides an insight into understanding which ISPs harbor a majority of P2P peers, which ones afford most transit to P2P flows, and possibly which ISPs should focus on anti-P2P policies the most. We observe an extremely skewed distribution, approximately 92 to 98% of P2P flows ending in tier 1 and tier 4 ISPs, and just 2 to 8% ending in tier 2 and 3 ISPs. We quantify the role of ISPs in allowing P2P flows to traverse through their domains and observe a similar skewed distribution wherein tier 1 and tier 4 ISPs contribute 92 to 95 % of all hops on most P2P flows. Moreover, we compare P2P traffic with http and Internet radio traces to uncover potential parameters to differentiate between these types of flows. We present detailed results based on active measurements taken over a 30 day period, spanning over half a million P2P flows, collected from measurements employing a number of popular P2P clients hosted inside two popular ISPs.

By integrating one or more (hard or soft) CPU core on the chip, new generation platform FPGAs have become configurable systems on a chip (CSoC) that support a combined software and hardware execution model. More recently, FPGAs, using new design tools, have also provided support for partial reconfiguration. The CSoC system designer is left with the task of interfacing IP Cores to the CPU and also for realizing partial reconfiguration across the cores. In this paper, we describe a software tool to automate the interface between the CPU and the reconfigurable fabric. Our tool generates hardware wrappers for the IP Cores that makes them look like a C function invocation in the source code. We also use our tool to support partial reconfiguration: the same wrapper is used for a multitude of IP Cores and the user selects the core to be invoked in the program.

Abstract—Low power, high performance sensor designs are of particular importance nowadays considering the plethora of emerging applications in a wide range of fields. Applications such as audio capture, FFT (Fast Fourier Transform), image sensing, motion detection, feature extraction and cepstrum calculations generate raw data in much larger quantity than more mundane applications as temperature and humidity sensing. In-situ processing of data is an effective way to reduce the amount of information needed to be transmitted over the wireless links. The Co-S platform which we develop addresses this very crucial aspect of sensor network design. The Co-S integrated with a System on Chip (SoC) based host platform and a gigabyte scale energy efficient data storage system, simultaneously satisfies the constraints of low power consumption and a small form factor. As opposed to the Sense-and-Send approach which simply entails transmission of raw sensory data from the nodes, the proposed Sense-and-Store paradigm, builds upon the availability of the low power flash memories for storage of sensed data along with key query results (max, min, average, extracted harmonics) for transmission through the network. This approach not only reduces energy consumption (typically two orders of magnitude), of the sensing node in question, but also intuitively reduces network traffic load vis-à-vis complete deployment scenario. Index Terms–Coprocessors, Performance, Flash Storage I.

The current trend of computationally intensive applications such as audio capture, FFT, image sensing, motion detection, feature extraction and cepstrum calculations being ported to the sensor domain, alludes towards the need to handle copious amounts of data in comparison to rudimentary temperature and humidity sensing applications. Developing a split architecture based sensor platform is an effective and efficient methodology to reach the crucial goal of in-situ processing of data. Resulting in a direct reduction in the amount of information needed to be transmitted over the wireless links. The Split-Node-Architecture (SNA) which we develop addresses this very crucial aspect of sensor network design. The suitability of employing the SNA architecture is demonstrated by its performance metrics obtained by running a 128point FFT benchmark. The higher throughput, of the SNA architecture enables faster data processing upto 24 times that of the Atmel AVR architecture (MICA). Additionally a 29% improvement in power efficiency vis a vis popular sensor platforms, tilts the scales unambiguously in favor of the SNA. The low power consumption of the Co-S module, a part of SNA, results in savings upto 6uJ per 128 point FFT operation when compared to the Stargate (Intel PXA 255).