Many systems for the parallel processing of big data are available today. Yet, few users can tell by intuition which system, or combination of systems, is "best" for a given workflow. Porting workflows between systems is tedious. Hence, users become "locked in", despite faster or more efficient systems being available. This is a direct consequence of the tight coupling between user-facing front-ends that express workflows (e.g., Hive, SparkSQL, Lindi, GraphLINQ) and the back-end execution engines that run them (e.g., MapReduce, Spark, PowerGraph, Naiad).

Many network functions executed in modern datacenters, e.g., load balancing, application-level QoS, and congestion control, exhibit three common properties at the data-plane: they need to access and modify state, to perform computations, and to access application semantics -- this is critical since many network functions are best expressed in terms of application-level messages. In this paper, we argue that the end hosts are a natural enforcement point for these functions and we present Eden, an architecture for implementing network functions at datacenter end hosts with minimal network support. Eden comprises three components, a centralized controller, an enclave at each end host, and Eden-compliant applications called stages. To implement network functions, the controller configures stages to classify their data into messages and the enclaves to apply action functions based on a packet's class. Our Eden prototype includes enclaves implemented both in the OS kernel and on programmable NICs. Through case studies, we show how application-level classification and the ability to run actual programs on the data-path allows Eden to efficiently support a broad range of network functions at the network's edge.

Modern data centers commonly run distributed applications that require low-latency communication, and whose performance is critical to service revenue. If as little as one machine in 10,000 is a latency outlier, around 18% of requests will experience high latency. The sacrifice of latency determinism for bandwidth, however, is not an inevitable one. In our R2D2 architecture, we conceptually split the data centre network into an unbuffered, unswitched low-latency network (LLNet) and a deeply buffered bandwidth centric network (BBNet). Through explicitly scheduling network multiplexing in software, our prototype implementation achieves 99.995% and 99.999% messaging latencies of 35us and 75us respectively for 1514-byte packets on a fully loaded network. Furthermore, we show that it is possible to merge the conceptually separate LLNet and BBNet networks onto the same physical infrastructure using commodity switched Ethernet hardware.

No abstract available.