Erasure coding is widely used in building reliable distributed object storage systems despite its high repair cost. Regenerating codes are a special class of erasure codes, which are proposed to minimize the amount of data needed for repair. In this paper, we assess how optimal repair can help to improve object storage systems, and we find that regenerating codes present unique challenges: regenerating codes repair at the granularity of chunks instead of bytes, and the choice of chunk size leads to the tension between streamed degraded read time and repair throughput.

Distributed file system has emerged in recent years as an efficient solution to store the large amount of data produced anytime and anywhere. In order to guarantee data reliability, it is necessary to introduce redundancy to the storage systems. Compared to simple replication, practical systems are increasingly adopting erasure codes for better storage efficiency. However, traditional erasure codes such as maximum-distance-separable (MDS) codes, are designed over a large finite field, which inevitably hinders the wide implementation of erasure codes. In this paper, we propose a new family of MDS codes with high computation efficiency. More specifically, only XOR operation is included in the encoding process to generate parity blocks. Upon failure of a storage node, we use the efficient Zigzag decoding method to recover the failed blocks, which achieves the optimal encoding and an efficient decoding. Furthermore, we implement the proposed codes in a distributed file system, and the results show the high performance of the new codes.

Recently, traditional erasure codes such as Reed-Solomon (RS) codes have been increasingly deployed in many distributed storage systems to reduce the large storage overhead incurred by the widely adopted replication scheme. However, these codes require significantly high resources with respect to network bandwidth and disk I/O during recovery of missing or unavailable data. It is referred as the recovery problem. In this paper, we dedicate to integrating exact minimum bandwidth regenerating codes into practical systems to solve the recovery problem. We design an implementation friendly storage code with the recently proposed BASIC framework and ZigZag decodable code for saving recovery bandwidth and disk I/O. We build a system called STORE based on this code and evaluate our prototype atop a HDFS cluster testbed with 21 nodes. As shown in this paper, the recovery bandwidth achieves minimum approximately during recovery of both data block and parity block with STORE. Another attractive result is that the recovery disk I/O also achieves minimum approximately during recovery of data block. Due to the reduction of recovery bandwidth and disk I/O, the degraded read throughput is boosted notably.

The explosion of big data stored in distributed file systems calls for more efficient storage paradigms. While replication is widely used to ensure data availability, erasure codes provide a much better tradeoff between storage and availability. Reed-Solomon (RS) codes are the standard design choice, however, their high repair cost is often considered an unavoidable price to pay for high storage efficiency and high reliability. BASIC codes can achieve the optimal tradeoff between storage capacity and repair bandwidth with much less complexity of regenerating codes, which is first proposed in [1]. This paper integrate one construction of the minimum storage BASIC (MS-BASIC) codes [2] into a Hadoop HDFS cluster testbed with up to 22 storage nodes. We demonstrate that MS-BASIC codes conform to the theoretical findings and achieve recovery bandwidth saving compared to the conventional recovery approach based on RS codes.