As the use of Apache Storm, a distributed real-time computation platform, becomes more widespread in community, scalability of the platform has become a key challenge. Currently, Apache Storm leverages Apache Zookeeper as a message broker for heartbeats and metrics. Using Apache Zookeeper in this manner may have been a good implementation choice at the inception of Apache Storm since it leverages the use of an existing proven open source technology instead of having to implement a proprietary message broker for Storm. However, using Apache Zookeeper in such a fashion has created one of the biggest bottlenecks in scaling Storm. In this paper, we first explore the key scalability issues that this architecture was creating as cluster and workload sizes grow at Yahoo. Next, we discuss in detail a new heartbeat and metrics service, called PaceMaker to address this scalability problem. Finally, we present experimental results to demonstrate that PaceMaker greatly relieves the workload and resource pressure on the ZooKeeper nodes which enables Storm to scale much further.

Summary form only given. Apache Storm is a popular low latency distributed stream processing framework. Apache Storm is used everywhere at Yahoo and at many other companies from automatically tagging every image uploaded to Flickr and analyzing trending search queries to monitoring production servers looking for problems. This hands-on tutorial is divided into two parts. The first part covers the basics of Storm, its architecture, and walks you through writing a simple application (not just word count). The second part looks more at how to modify Storm and will walk you through adding in a new feature.

The initial design of Apache Hadoop [1] was tightly focused on running massive, MapReduce jobs to process a web crawl. For increasingly diverse companies, Hadoop has become the data and computational agorá---the de facto place where data and computational resources are shared and accessed. This broad adoption and ubiquitous usage has stretched the initial design well beyond its intended target, exposing two key shortcomings: 1) tight coupling of a specific programming model with the resource management infrastructure, forcing developers to abuse the MapReduce programming model, and 2) centralized handling of jobs' control flow, which resulted in endless scalability concerns for the scheduler.