Most supercomputers nowadays are based on large clusters, which call for sophisticated, scalable, and decentralized metadata processing techniques. From the perspective of maximizing metadata throughput, an ideal metadata distribution policy should automatically balance the namespace locality and even distribution without manual intervention. None of existing metadata distribution schemes is designed to make such a balance. We propose a novel metadata distribution policy, Dynamic Dir-Grain (DDG), which seeks to balance the requirements of keeping namespace locality and even distribution of the load by dynamic partitioning of the namespace into size-adjustable hierarchical units. Extensive simulation and measurement results show that DDG policies with a proper granularity significantly outperform traditional techniques such as the Random policy and the Subtree policy by 40 percent to 62 times. In addition, from the perspective of file system reliability, metadata consistency is an equally important issue. However, it is complicated by dynamic metadata distribution. Metadata consistency of cross-metadata server operations cannot be solved by traditional metadata journaling on each server. While traditional two-phase commit (2PC) algorithm can be used, it is too costly for distributed file systems. We proposed a consistent metadata processing protocol, S2PC-MP, which combines the two-phase commit algorithm with metadata processing to reduce overheads. Our measurement results show that S2PC-MP not only ensures fast recovery, but also greatly reduces fail-free execution overheads.

In large-scale cluster systems, the failure rate of network connection is non-negligibly high. A cluster file system must have the ability to handle network failures in order to provide high-available data accesses service. Traditionally, network failure handling is only guaranteed by network protocol, or implemented within the file system semantic layer. We present the high-available message-passing framework which is called HMF. Based on the operation hierarchy in cluster file system, HMF guarantees the availability of each pair of network transmissions and their interaction with the file system sub-operations. It separates the network fault-tolerance design from the file system and keeps a simple interface between them. HMF could handle a lot of network failures internally, which greatly simplifies the implementation of file system semantic layer. Performance results show that HMF can increase the availability of message passing and reduce the cost of recovery from network failures. When there are two network channels, HMF also improves aggregate I/O bandwidth by 80% in normal condition while the performance degradation due to recovery is below 10%.

Data availability is a challenging issue for large-scale cluster file systems built upon thousands of individual storage devices. Replication is a well-known solution used to improve data availability. However, how to efficiently guarantee replicas consistency under concurrent conflict mutations remains a challenge. Moreover, how to quickly recover replica consistency from a storage server crash or storage device failure is also a tough problem. In this paper, we present a replication-based data availability mechanism designed for a large-scale cluster file system prototype named LionFS. Unlike other replicated storage systems that serialize replica updates, LionFS introduces a relaxed consistency model to enable concurrent updating all replicas for a mutation operation, greatly reducing the latency of operations. LionFS ensures replica consistency if applications use file locks to synchronize the concurrent conflict mutations. Another novelty of this mechanism is its light-weight log, which only records failed mutations and imposes no overhead on failure-free execution and low overhead when some storage devices are unavailable. Furthermore, recovery of replica consistency needs not stop the file system services and running applications. Performance evaluation shows that our solution achieves 50–70% higher write performance than serial replica updates. The logging overhead is shown to be low, and the recovery time is proportional to the amount of data written during the failure.

How to distribute the items in the file system hierarchy across a group of metadata servers is an important issue that determines the holistic metadata processing performance (HMPP) of a cluster file system which manages its metadata by a group of metadata servers. The HMPP is affected by two factors: balance degree of metadata distribution and number of branch points. Two types of well-used metadata distribution policies are the dynamic subtree policy and the random policy. Both of them emphasize one factor and neglect the other factor. As a result, their HMPP is low. In order to make good use of processing capacity of all metadata servers, we present a novel metadata distribution policy, called dynamic dir-grain (DDG) policy, which takes both factors into account. Our performance results show that this policy is potentially more efficient than the other two types of policies under real environments, as well as the conditions of creation or removal of a large hierarchy

For its simplicity, reliability and maturity, NFS is widely-used in clusters. However, due to its high overheads and implementation limitations, the standard NFS cannot fully exert the potential abilities provided by multiple network channels and multiple SCSI channels on the server. In this paper, we present a new efficient way to high performance NFS implementation for cluster applications. By adding mechanisms to make good use of NFS server's multiple communication channels and multiple I/O channels, CluserNFS can potentially provide better I/O performance than standard NFS, as illustrated by our simulation experiment results.

Extendible hashing is a kind of fast indexing technology; it provides with a way of storing structural data records so that each of them can be gotten very quickly. In this paper, we present our own specially optimized implementation of using extendible hashing in cluster file system is order to support large directory.