Apache Kafka is a scalable publish-subscribe messaging system with its core architecture as a distributed commit log. It was originally built at LinkedIn as its centralized event pipelining platform for online data integration tasks. Over the past years developing and operating Kafka, we extend its log-structured architecture as a replicated logging backbone for much wider application scopes in the distributed environment. In this abstract, we will talk about our design and engineering experience to replicate Kafka logs for various distributed data-driven systems at LinkedIn, including source-of-truth data storage and stream processing.

Linked In is among the largest social networking sites in the world. As the company has grown, our core data sets and request processing requirements have grown as well. In this paper, we describe a few selected data infrastructure projects at Linked In that have helped us accommodate this increasing scale. Most of those projects build on existing open source projects and are themselves available as open source. The projects covered in this paper include: (1) Voldemort: a scalable and fault tolerant key-value store, (2) Data bus: a framework for delivering database changes to downstream applications, (3) Espresso: a distributed data store that supports flexible schemas and secondary indexing, (4) Kafka: a scalable and efficient messaging system for collecting various user activity events and log data.

Virtual machines (VM) are promising as system software in sensor networks. A major impediment to their widespread acceptance is their performance over- head. Compiling VM bytecode to native code addresses this, but increases footprint and code distribution costs. Thus, there is an important tradeoff between cost of computing and cost of communication due to code distri- bution. In this paper, we describe a remote Just-In-Time (JIT) compilation service that is effective in combining interpretation with native execution to arrive at an efficient hybrid execution configuration. The principles apply to any middleware used to develop applications in sensor networks.

Sensor networks are being deployed at massive scales, containing a range of platforms. Programming paradigms for sensor networks should meet the attendant challenges of scale and heterogeneity. Researchers have considered virtual machines as a means to address these challenges. However, in order to satisfy the resource limitations of sensor nodes, they export only a minimal set of services to the application programmer. This makes applications of even moderate complexity difficult to implement. We present VMSTAR --- a framework for building resource-efficient virtual machines that scale and export comprehensive service suites on a per-application basis. We advocate the use of fine-grained software synthesis to build resource-efficient system software, and facilitate both application changes and system software upgrades at runtime through an efficient incremental update scheme. We have used our framework to build virtual machines on the Mica platform and describe how virtual machines are effective in meeting the difficult demands of heterogeneity and reprogrammability.