In cloud ecosystems, distributed block storage systems are used to provide a persistent block storage service, which is the fundamental building block for operating cloud native services. However, existing distributed storage systems performed poorly for random write workloads in an all-NVMe storage configuration, becoming CPU-bottlenecked. Our roofline-based approach to performance analysis on a conventional distributed block storage system with NVMe SSDs reveals that the bottleneck does not lie in one specific software module, but across the entire software stack; (1) tightly coupled I/O processing, (2) inefficient threading architecture, and (3) local backend data store causing excessive CPU usage. To this end, we re-architect a modern distributed block storage system for improving random write performance. The key ingredients of our system are (1) decoupled operation processing using non-volatile memory, (2) prioritized thread control, and (3) CPU-efficient backend data store. Our system emphasizes low CPU overhead and high CPU efficiency to efficiently utilize NVMe SSDs in a distributed storage environment. We implement our system in Ceph. Compared to the native Ceph, our prototype system delivers more than 3x performance improvement for small random write I/Os in terms of both IOPS and latency by efficiently utilizing CPU cores.

Storage I/O performance remains a key factor that determines the overall user experience of a computer system. This is especially true for mobile systems as users commonly browse and navigate through many high-quality pictures and video clips stored in their device. The appetite for more appealing user interface has continuously pushed the mobile storage interface speed up; emerging UFS 2.0 standard provisions a maximum bandwidth of as high as 1,200 MB/s. In this work, we propose, design, and implement a mobile storage architecture that leverages the high-speed DRAM interface for communication, thus substantially expanding the storage performance headroom. In order to effectively turn the existing DRAM interface into a storage interface, we design a new storage protocol that runs on top of the DRAM interface. Our protocol builds on a small host interface buffer structure mapped to the system's memory space. Based on this protocol, we develop and fabricate a storage controller chip that natively supports the LPDDR3 interface. We also develop a host software stack (Linux device driver and boot loader) and a host platform board. Finally we show the feasibility of our proposal by constructing a full Android system running on the developed storage device and platform. Our detailed evaluation shows that the proposed storage architecture has very low protocol handling overheads and compares favorably to a UFS 2.0 device. The proposed architecture obviates the need for implementing a separate host-side storage controller on a mobile CPU chip.