We introduce split-level I/O scheduling, a new framework that splits I/O scheduling logic across handlers at three layers of the storage stack: block, system call, and page cache. We demonstrate that traditional block-level I/O schedulers are unable to meet throughput, latency, and isolation goals. By utilizing the split-level framework, we build a variety of novel schedulers to readily achieve these goals: our Actually Fair Queuing scheduler reduces priority-misallocation by 28x; our Split-Deadline scheduler reduces tail latencies by 4x; our Split-Token scheduler reduces sensitivity to interference by 6x. We show that the framework is general and operates correctly with disparate file systems (ext4 and XFS). Finally, we demonstrate that split-level scheduling serves as a useful foundation for databases (SQLite and PostgreSQL), hypervisors (QEMU), and distributed file systems (HDFS), delivering improved isolation and performance in these important application scenarios.

Explosion in Big Data has led to a surge in extremely large-scale Big Data analytics platforms, resulting in burgeoning energy costs. Big Data compute model mandates strong data-locality for computational performance, and moves computations to data. State-of-the-art cooling energy management techniques rely on thermal-aware computational job placement/migration and are inherently data-placement-agnostic in nature. T* takes a novel, data-centric approach to reduce cooling energy costs and to ensure thermal-reliability of the servers. T* is cognizant of the uneven thermal-profile and differences in thermal-reliability-driven load thresholds of the servers, and the differences in the computational jobs arrival rate, size, and evolution life spans of the Big Data placed in the cluster. Based on this knowledge, and coupled with its predictive file models and insights, T* does proactive, thermal-aware file placement, which implicitly results in thermal-aware job placement in the Big Data analytics compute model. Evaluation results with one-month long real-world Big Data analytics production traces from Yahoo! show up to 42% reduction in the cooling energy costs with T* courtesy of its lower and more uniform thermal-profile and 9x better performance than the state-of-the-art data-agnostic cooling techniques.

We present a detailed evaluation and sensitivity analysis of an energy-conserving, highly scalable variant of the Hadoop Distributed File System (HDFS) called Green-HDFS. Green HDFS logically divides the servers in a Hadoop cluster into Hot and Cold Zones and relies on insightful data-classification driven energy-conserving data placement to realize guaranteed, substantially long periods(several days) of idleness in a significant subset of servers in the Cold Zone. Detailed lifespan analysis of the files in a large-scale production Hadoop cluster at Yahoo! points at the viability of Green HDFS. Simulation results with real-worldYahoo! HDFS traces show that Green HDFS can achieve 24% energy cost reduction by doing power management in only one top-level tenant directory in the cluster and meets all the scale-down mandates in spite of the unique scale-down challenges present in a Hadoop cluster. If Green HDFS technique is applied to all the Hadoop clusters at Yahoo! (amounting to 38000 servers), $2.1millioncan be saved in energy costs per annum. Sensitivity analysis shows that energy-conservation is minimally sensitive to the thresholds in Green HDFS. Lifespan analysis points out that one-size-fits-all energy-management policies won'tsuffice in a multi-tenant Hadoop Cluster.

The objective of this research is to present an energy-conserving, self-adaptive Commodity Green Cloud Storage, called Lightning. Lightning's File System dynamically configures the servers in the Cloud Storage into logical Hot and Cold Zones. Lightning uses data-classification driven data placement to realize guaranteed, substantially long, periods (several days) of idleness in a significant subset of servers designated as the Cold Zone, in the commodity datacenter backing the Cloud Storage. These servers are then transitioned to inactive power modes and the resulting energy savings substantially reduce the operating costs of the datacenter. Furthermore, the energy savings allow Lightning to improve the data access performance by incorporation of high-performance, though high-cost Solid State Drives (SSD) without exceeding the total cost of ownership (TCO) of the datacenter. Analytical cost model analysis of Lightning suggests savings in the upwards of $24 million in the TCO of a 20,000 server datacenter. The simulation results show that Lightning can achieve 46% energy costs reduction even when the datacenter is at 80% capacity utilization.

Bug reproduction is critically important for diagnosing a production-run failure. Unfortunately, reproducing a concurrency bug on multi-processors (e.g., multi-core) is challenging. Previous techniques either incur large overhead or require new non-trivial hardware extensions.