In this work, we develop the Orchestrated File System (OrcFS) for Flash storage. OrcFS vertically integrates the log-structured file system and the Flash-based storage device to eliminate the redundancies across the layers. A few modern file systems adopt sophisticated append-only data structures in an effort to optimize the behavior of the file system with respect to the append-only nature of the Flash memory. While the benefit of adopting an append-only data structure seems fairly promising, it makes the stack of software layers full of unnecessary redundancies, leaving substantial room for improvement. The redundancies include (i) redundant levels of indirection (address translation), (ii) duplicate efforts to reclaim the invalid blocks (i.e., segment cleaning in the file system and garbage collection in the storage device), and (iii) excessive over-provisioning (i.e., separate over-provisioning areas in each layer). OrcFS eliminates these redundancies via distributing the address translation, segment cleaning (or garbage collection), bad block management, and wear-leveling across the layers. Existing solutions suffer from high segment cleaning overhead and cause significant write amplification due to mismatch between the file system block size and the Flash page size. To optimize the I/O stack while avoiding these problems, OrcFS adopts three key technical elements. First, OrcFS uses disaggregate mapping , whereby it partitions the Flash storage into two areas, managed by a file system and Flash storage, respectively, with different granularity. In OrcFS, the metadata area and data area are maintained by 4Kbyte page granularity and 256Mbyte superblock granularity. The superblock-based storage management aligns the file system section size, which is a unit of segment cleaning, with the superblock size of the underlying Flash storage. It can fully exploit the internal parallelism of the underlying Flash storage, exploiting the sequential workload characteristics of the log-structured file system. Second, OrcFS adopts quasi-preemptive segment cleaning to prohibit the foreground I/O operation from being interfered with by segment cleaning. The latency to reclaim the free space can be prohibitive in OrcFS due to its large file system section size, 256Mbyte. OrcFS effectively addresses this issue via adopting a polling-based segment cleaning scheme. Third, the OrcFS introduces block patching to avoid unnecessary write amplification in the partial page program. OrcFS is the enhancement of the F2FS file system. We develop a prototype OrcFS based on F2FS and server class SSD with modified firmware (Samsung 843TN). OrcFS reduces the device mapping table requirement to 1/465 and 1/4 compared with the page mapping and the smallest mapping scheme known to the public, respectively. Via eliminating the redundancy in the segment cleaning and garbage collection, the OrcFS reduces 1/3 of the write volume under heavy random write workload. OrcFS achieves 56% performance gain against EXT4 in varmail workload.

In this work, we develop an intelligent storage system framework for soft real-time applications. Modern software systems consist of a collection of layers and information exchange across the layers is performed via well-defined interfaces. Due to the strictness and inflexibility of interface definition, it is not possible to pass the information specific to one layer to other layers. In practice, the exploitation of this information across the layers can greatly enhance the performance, reliability, and manageability of the system. We address the limitation of legacy interface definition via enabling intelligence in the storage system. The objective is to enable the lower-layer entity, for example, a physical or block device, to conjecture the semantic and contextual information of that application behavior which cannot be passed via the legacy interface. Based upon the knowledge obtained by the intelligence module, the system can perform a number of actions to improve the performance, reliability, security, and manageability of the system. Our intelligence storage system focuses on optimizing the I/O subsystem performance for a soft real-time application. Our intelligence framework consists of three components: the workload monitor , workload analyzer , and system optimizer . The workload monitor maintains a window of recent I/O requests and extracts feature vectors in regular intervals. The workload analyzer is trained to determine the class of the incoming workload by using the feature vector. The system optimizer performs various actions to tune the storage system for a given workload. We use confidence rate boosting to train the workload analyzer. This sophisticated learner achieves a higher than 97% accuracy of workload class prediction. We develop a prototype intelligence storage system on the legacy operating system platform. The system optimizer performs; (1) dynamic adjustment of the file-system-level read-ahead size; (2) dynamic adjustment of I/O request size; and (3) filtering of I/O requests. We examine the effect of this autonomic optimization via experimentation. We find that the storage level pro-active optimization greatly enhances the efficiency of the underlying storage system. The sophisticated intelligence module developed in this work does not restrict its usage for performance optimization. It can be effectively used as classification engine for generic autonomic computing environment, i.e. management, diagnosis, security and etc.