Vectorized execution engines process large datasets by decomposing computations into concise (tight) loops, which can be more efficiently executed by modern hardware. Providing loops that are optimal for execution usually adds burden to the software development process, as developers are required to understand details of vectorized execution, columnar data layout, data encodings, and the code compilation process itself, presenting a steep learning curve and challenges to organizations building and scaling large engineering teams. Due to their large quantity, scalar function authoring accentuates this problem. In our experience building the Velox open source execution engine, we have observed that exposing a large number of developers to the complexity inherent to vectorization resulted in a disproportionate amount of bugs and performance inefficiencies. In this paper, we describe the simple function interface (SFI) created to address this issue. SFI highly simplifies scalar function authoring by encapsulating the vectorization complexity required to generate tight loops, and presenting developers with a simpler, conciser, and more natural row-based interface - without sacrificing performance. SFI also hides columnar layout details, while providing developers the flexibility to efficiently implement advanced features such as functions with nested and recursive parameter types, type variables, variadic parameters, and generic types. Today, more than a thousand functions have been added to Velox using the SFI, implementing popular open source SQL dialects and internal domain-specific use cases at Meta, and are in active production use. While this paper presents implementation details, performance pitfalls, experimental results, and our overall experience developing the state-of-the-art Velox vectorized execution engine, we believe the concepts and trade-offs to be fundamentally equivalent and generally applicable to other vectorized engines.

Blue Whale File System (BWFS) is a distributed file system with proven high performance and high scalability. In order to provide high reliability for BWFS, we designed and implemented a new architecture named BW-FILERAID, which combines distributed file system well with the RAID technology. BW-FILERAID uses client-driven file-level RAID to satisfy the requirement of reliability and optimizes the RAID performance by utilizing the file system information. The schemes of Dynamic Stripe Allocation (DSA) and Asynchronous Parity Calculating (APC) are proposed to reduce the impact of small write performance problem which is incurred by classic RAID5 implementation. This paper introduces and evaluates BW-FILERAID. As compared with the Linux software RAID5 systems, BW-FILERAID can achieve significantly higher WRITE performance which increases by 8.2~25.9% while with the same READ performance as RAID0. At last, BW-FILERAID also has very good system scalability.

The paper presents a new dynamic mapping virtual storage system, which is characterized by: (a) distribute storage resources according to need, thus improving storage resource utilization, (b) more fundamentally, through write-anywhere pattern to sequentialize concurrent write requests by dynamic address mapping mechanism, thus improving concurrent write performance for applications. Meanwhile, taking into account imperfection small size write requests handling and the negative impact on read performance, we combine the system with CBD system which is responsible for optimizing cache management. We have implemented our virtual storage system in Linux kernel 2.6.18 as a pseudo device driver. The experiment results show that concurrent write bandwidth approaches raw disk write performance. And in 64-stream concurrent write case, E-ASD can outperform concurrent write bandwidth of LVM by up to 160%, and its max single-stream read loss is less than 25% compared to LVM, with concur-rent read performance close to each other. Read performance still has space to be improved given larger cache and proper prefetching.

This paper analyzed the performance characteristics of classic 2Q algorithm when it was performed on mail-service workloads, and proposes an improved algorithm, called 2Q*. The simulation results show that 2Q* algorithm can outperform the other replacement algorithms, including the classic 2Q algorithm, for all the cache sizes and various mail-service workloads. To verify the simulation results in real system, we integrated the algorithm into FlexiCache, a partitioned buffer cache system, and joined it with a popular adaptive sequential pre-fetch policy properly. The experiment results verify the effectiveness of 2Q* algorithm for mail service applications. By joint with the pre-fetch policy, the performance is further improved. Moreover, its runtime overhead is also fairly low.

Scrubbing is an efficient tool to reduce latent sector errors for long-term storage. There are two established models to evaluate the effect of scrubbing. One is attributed sector errors to the contact of head and disk surface. Accordingly, sector errors are strongly related to the amount of I/O operations. The other model views the sector errors as the consequence of operating conditions. However, the sector errors in most cases are the combination of both factors. In this paper, our study analyzes, compares both methods and develops our new model basing on the average probability of both factors. Our result demonstrates that though the probability of chronic sector corruption with environmental factors is actually smaller than the probability of sector error due to I/O operations for a tagged sector, the contribution of the former to the overall sector errors is more or less equal to the latter and will ultimately exceed the latter with time increase. Furthermore, we carry out a new implementation of the deterministic scheme in our virtualization platform for long-term backup application, basing on our analysis and the locality of sector errors.

This paper presents a flexible two-layer buffer caching scheme to improve the performance of storage cache which is used to serve multiple concurrently accessing applications with diverse access patterns. To achieve this, the proposed scheme dynamically partitions the cache among applications. At the first layer, it uses a configurable global cache allocation policy to make adaptive cache allocations in response to the evolving access patterns of competing applications. In contrast to traditional global replacement approach, the allocation policy adopted in our scheme utilizes applicationpsilas marginal utility but not cache demand for allocation, so as to minimize the total number of cache misses. At the second laver, our scheme tries to maximize the utilization of allocated cache blocks by applying each application with an appropriate local replacement algorithm based on its access pattern. We have implemented our scheme in Linux kernel 2.6.18 as a pseudo device driver and measured its performance using various real-life workloads. The experiment results show that compared with Linux page cache, the proposed scheme can reduce the overall response time by up to 2 times with an average of 50% and reduce the overall disk load by up to 31% with an average of 21%.

This paper presents a symbolic AC sensitivity analysis technique using a graph-reduction based symbolic simulator GRASS. Symbolic sum-of-products are derived from a graph reduction process and represented by a binary decision diagram. The GRASS simulator maintains a one-to-one correspondence between the circuit parameters and the simulator symbols, with which differentiating the frequency response with respect to any circuit parameter becomes straightforward. The implementation details of symbolic AC sensitivity are presented and the potential applications are demonstrated.

With popularity of different kind of search engines on WWW, it requires the backend storage system to provide better physical I/O performance to speedup the query service perceived by end users. However, existing general purpose designed replacement algorithm canpsilat performs well for the web search applications. This paper first studies the access pattern of various real-life web search workload and then propose a new replacement algorithm RED-LRU based on the observed access properties. The simulation results shows that our proposed algorithm uniformly outperform the other replacement algorithms for all the workloads and cache size. To validate the simulation results, we integrate RED-LRU algorithm into a real storage cache DPCache. The experiment results in real system confirm the effectiveness of our proposed algorithm in improving the caching performance for web search application. Moreover, the runtime overhead of RED-LRU is also fairly low in practice.

P-Cache to provide prioritized caching service for storage server which is used to serve multiple concurrently accessing applications with diverse access patterns and unequal importance. Given the replacement algorithm and the application access patterns, the end performance of each individual application in a shared cache is actually determined by its allocated cache resource. So, P-Cache adopts a dynamic partitioning approach to explicitly divide cache resource among applications and utilizes a global cache allocation policy to make adaptive cache allocations to guarantee the preset relative caching priority among competing applications. We have implemented P-Cache in Linux kernel 2.6.18 as a pseudo device driver and measured its performance using synthetic benchmark and real-life workloads. The experiment results show that the prioritized caching service provided by P-Cache can not only be used to support application priority but can also be utilized to improve the overall storage system performance. Its runtime overhead is also smaller compared with Linux page cache.

Scalability, which indicates the capacity of system to support the maximal number of clients, is very important to cluster file system. But in practical applications, the bandwidth that storage devices provide doesnpsilat linearly grow with the increasing number of clients which leads to bad QoS and restricts the scalability of cluster file system. This paper analyses the bottleneck of scalability of Blue Whale Cluster File System (BWFS) in video-editing application, and present a solution that clients employ the spare disk resource to cache data from remote server. By careful chosen of cache strategy and replacement algorithm, clients can access remote data from local disk, which significantly reduce random access to server and efficiently reduce the workload of storage servers. Therefore, the scalability of the whole system is significantly enhanced. We also implement a prototype to validate the solution and the result shows that it can enhance the scalability of the Blue Whale System by 80% at most.

We have designed and implemented the blue whale file system (BWFS), a scalable distributed file system for large distributed data-intensive applications. With many of the features as previous distributed file systems, BWFS has successfully met our storage needs and is widely deployed within many fields. Although excellent for high-bandwidth access to large files, BWFS's out-of-band data transfer mode provides low efficiency under small files intensive workloads. In order to improve the overall performance of the file system, we propose a novel data transfer scheme. In such novel scheme, BWFS transfers data with the hybrid data transfer policy that small files are transferred with in-band mode while large files are transferred with out-of-band mode. The prototype design and implementation is described and the various experiments are presented to demonstrate that the significant performance benefits of our prototype implementation under the small files intensive workloads. For small files intensive applications, BWFS can achieve significantly higher throughput which increases by 60%.

The paper presents a novel network RAID storage system based on the out-of-band virtualization architecture and the backend centralized redundant management. The application servers can inquire the mapping information of virtual disk from the out-of-band virtualization server and directly access the storage nodes. The read request can fetch the data from special storage node, while the write request is not only stored into the storage node, and also mirrored into the redundant server by the storage node. The redundant server can cache updated data in local disks with log-structured mode, and calculate the parity of RAID5 in the background process when the system is idle. It relieves the bottle problem of I/O performance and low reliability danger of single controller in the front-end centralized management system. And the layout of RAID1/RAID5 on the data block has acquired the trade-off among performance, reliability and cost.