Recent developments in non-volatile memory and networking technologies raise new challenges and opportunities for architecting storage systems. In this paper we propose Telepathy, a lightweight data access protocol for NVRAM+RDMA-based distributed storage systems. Telepathy is a fully distributed protocol whose I/O writes can be coordinated by any server node, and I/O reads can be served by any of the replicas. Telepathy guarantees strongly-consistent reads while providing high I/O concurrency. Hybrid RDMA operations are used to transmit data directly and efficiently to the NVRAM of target servers. The correctness of Telepathy is verified with a formal proof of consistency, and its performance is validated with YCSB benchmarks on the Chameleon cluster. Telepathy can achieve low I/O latencies and high throughput, with low CPU utilization.

Advances in persistent memory and networking hardware are changing the architecture of storage systems and data management services in datacenters. Distributed, one-sided RDMA access to memory-resident data shows tremendous improvements in throughput, latency and server CPU utilization of storage servers. However, the silent nature of one-sided I/O simultaneously creates new challenging problems for providing QoS in such systems. In this paper, we propose Haechi, a work-conserving, token-based QoS mechanism to guarantee reservations and limits in storage systems that provide one-sided I/O services. Haechi decouples QoS enforcement into a QoS engine at the client and a QoS monitor at the data node. It leverages adaptive token dispatch, token conversion, and silent I/O reporting to guarantee the reservations of distributed clients while maintaining high server utilization. Empirical evaluations on the Chameleon cluster, with different reservation distributions and I/O access patterns, show that Haechi is successful in providing differentiated QoS with negligible overhead for token management.

Advances in storage, processing, and networking hardware are changing the structure of distributed applications. RDMA networks provide multiple communication mechanisms that enable novel hybrid protocols specialized to different data transfer requirements. In this paper, we present a distributed communication scheme that separates control and data communication channels directly at the RNIC rather than the application level. We develop a new communication artifact, a remote random access buffer, to efficiently implement this separation. Data messages are sent silently to the receiver, which is informed of the location of the data by a subsequent control message. Experiments on an RDMA-enabled cluster with micro benchmarks and two distributed applications validate the performance benefits of our approach.

The paper addresses the problem of providing performance QoS guarantees in a clustered storage system. Multiple related storage objects are grouped into logical containers called buckets, which are distributed over the servers based on the placement policies of the storage system. QoS is provided at the level of buckets. The service credited to a bucket is the aggregate of the IOs received by its objects at all the servers. The service depends on individual time-varying demands and congestion at the servers. We present a token-based, coarse-grained approach to providing IO reservations and limits to buckets. We propose pShift, a novel token allocation algorithm that works in conjunction with token-sensitive scheduling at each server to control the aggregate IOs received by each bucket on multiple servers. pShift determines the optimal token distribution based on the estimated bucket demands and server IOPS capacities. Compared to existing approaches, pShift has far smaller overhead, and can be accelerated using parallelization and approximation. Our experimental results show that pShift provides accurate QoS among the buckets with different access patterns, and handles runtime demand changes well.

Providing latency support is an important problem for clustered storage systems. In this paper, we present Fair-EDF, a framework for latency guarantees in shared storage servers. It provides fairness control while supporting latency guarantees. Fair-EDF extends the pure earliest deadline first (EDF) scheduler by adding a controller to shape the workloads. Under overload it selects a minimal number of requests to drop and to choose the dropped requests in a fair manner. The evaluation results show Fair-EDF provides steady fairness control among a set of clients with different runtime behaviors.

Emerging nonvolatile RAM (NVRAM) technologies have a limit on the number of writes that can be made to any cell, similar to the erasure limits in NAND Flash. This motivates the need for wear leveling techniques to distribute the writes evenly among the cells. Unlike NAND Flash, cells in NVRAM can be rewritten without the need for erasing the entire containing block, avoiding the issues of space reclamation and garbage collection, motivating alternate approaches to the problem. In this article, we propose a hierarchical wear-leveling model called Ouroboros wear leveling. Ouroboros uses a two-level strategy whereby frequent low-cost intraregion wear leveling at small granularity is combined with interregion wear leveling at a larger time interval and granularity. Ouroboros is a hybrid migration scheme that exploits correct demand predictions in making better wear-leveling decisions while using randomization to avoid wear-leveling attacks by deterministic access patterns. We also propose a way to optimize wear-leveling parameter settings to meet a target smoothness level under limited time and space overhead constraints for different memory architectures and trace characteristics. Several experiments are performed on synthetically generated memory traces with special characteristics, two block-level storage traces, and two memory-line-level memory traces. The results show that Ouroboros wear leveling can distribute writes smoothly across the whole NVRAM with no more than 0.2% space overhead and 0.52% time overhead for a 512GB memory.