Deduplication is a commonly-used technique on disk-based storage pools. However, deduplication has not been used for tape-based pools: tape characteristics, such as high mount and seek times combined with data fragmentation resulting from deduplication create a toxic combination that leads to unacceptably high retrieval times. This work proposes DedupT, a system that efficiently supports deduplication on tape pools. This paper (i) details the main challenges to enable efficient deduplication on tape libraries, (ii) presents a class of solutions based on graph-modeling of similarity between data items that enables efficient placement on tapes; and (iii) presents the design and evaluation of novel cross-tape and on-tape chunk placement algorithms that alleviate tape mount time overhead and reduce on-tape data fragmentation. Using 4.5 TB of real-world workloads, we show that DedupT retains at least 95% of the deduplication efficiency. We show that DedupT mitigates major retrieval time overheads, and, due to reading less data, is able to offer better restore performance compared to the case of restoring non-deduplicated data.

Very large block-level data backup systems need scalable data deduplication and garbage collection techniques to make efficient use of the storage space and to minimize the performance overhead of doing so. Although the deduplication and garbage collection logic is conceptually straight-forward, their implementations pose a significant technical challenge because only a small portion of their associated data structures could fit into memory. In this paper, we describe the design, implementation and evaluation of a data deduplication and garbage collection engine called Sungem that is designed to remove duplicate blocks in incremental data backup streams. Sungem features three novel techniques to maximize the deduplication throughput without compromising the deduplication ratio. First, Sungem puts related fingerprint sequences, rather than fingerprints from the same backup stream, into the same container in order to increase the fingerprint prefetching efficiency. Second, to make the most of the memory space reserved for storing fingerprints, Sungem varies the sampling rates for fingerprint sequences based on their stability. Third, Sungem combines reference count and expiration time in a unique way to arrive at the first known incremental garbage collection algorithm whose bookkeeping overhead is proportional to the size of a disk volume's incremental backup snapshot rather than its full backup snapshot. We evaluated the Sungem prototype using a real-world data backup trace, and showed that the average throughput of Sungem is more than 200,000 fingerprint lookups per second on a standard X86 server, including the garbage collection cost.

Traditional storage systems provide a simple read/write interface, which is inadequate for low-locality update-intensive workloads because it limits the disk scheduling flexibility and results in inefficient use of buffer memory and raw disk bandwidth. This paper describes an update-aware disk access interface that allows applications to explicitly specify disk update requests and associate with such requests call-back functions that will be invoked when the requested disk blocks are brought into memory. Because call-back functions offer a continuation mechanism after retrieval of requested blocks, storage systems supporting this interface are given more flexibility in scheduling pending disk update requests. In particular, this interface enables a simple but effective technique called Batching mOdifications with Sequential Commit (BOSC), which greatly improves the sustained throughput of a storage system under low-locality update-intensive workloads. In addition, together with a space-efficient low-latency disk logging technique, BOSC is able to deliver the same durability guarantee as synchronous disk updates. Empirical measurements show that the random update throughput of a BOSC-based B+ tree is more than an order of magnitude higher than that of the same B+ tree implementation on a traditional storage system.

Virtualization provides the possibility of whole machine migration and thus enables a new form of fault tolerance that is completely transparent to applications and operating systems. The most seamless virtualization-based fault tolerance configuration is an active/active master-slave configuration, in which the memory states of the master and slave virtual machine are periodically synchronized and the slave can immediately take over when the master dies without losing any on-going connections. The frequency of memory state synchronization has a direct impact on the performance overhead, the application response time, and the fail-over delay. This paper describes a speculative memory state synchronization technique that could effectively reduce the synchronization frequency without increasing the performance overhead, and presents a comprehensive performance study of these techniques under three realistic workloads, the TPC-E benchmark, the SPECsfs 2008 CIFS benchmark, and a Microsoft Exchange workload. We show that the proposed technique can effectively cut down the amount of memory state synchronization traffic by more than an order of magnitude.

There has been increasing interest in deploying data reduction techniques in primary storage systems. This paper analyzes large datasets in four typical enterprise data environments to find patterns that can suggest good design choices for such systems. The overall data reduction opportunity is evaluated for deduplication and compression, separately and combined, then in-depth analysis is presented focusing on frequency, clustering and other patterns in the collected data. The results suggest ways to enhance performance and reduce resource requirements and system cost while maintaining data reduction effectiveness. These techniques include deciding which files to compress based on file type and size, using duplication affinity to guide deployment decisions, and optimizing the detection and mapping of duplicate content adaptively when large segments account for most of the opportunity.

A powerful functionality enabled by modern virtualization technologies is the ability to move a virtual machine (VM) from one physical machine to another, which enables unprecedented flexibility for system fault tolerance and load balancing. However, no similar capability exists for physical machines. This paper describes the first known successful implementation of migrating a physical machine's state from one physical Linux machine to another. This physical machine state migration (PMSM) capability greatly decreases the amount of disruption due to scheduled shut-down for non-virtualized physical machines, and is more challenging than VM migration because it cannot rely on a separate piece of software to perform the state transfer, e.g., the hyper visor in the case of VM migration. The PMSM prototype described in this paper is adapted from Linux's hibernation facility. The current PMSM prototype can migrate a physical machine running the MySQL DBMS server under 7 seconds.

A block-level continuous data protection (CDP) system logs every disk update to a network storage server it protects, so as to support more flexible recovery time objective (RTO) and recovery point objective (RPO). To provide efficient access to historical snapshots, block-level CDP systems maintain multiple index structures, each of which needs to be updated whenever a disk block update operation is logged. Because these index structures are too large to be held in memory, updating their on-disk versions in real time becomes a major performance bottleneck that prevents existing CDP systems from scaling to large data backup applications. This paper describes the design and implementation of a high-performance index update mechanism that logs index updates, batches them in memory, and commits them using mostly sequential disk I/O. Sequential commit greatly reduces the cost of bringing in and writing back each on-disk index page. Update batching further amortizes this cost over multiple index update operations. Empirical performance measurements demonstrate that the proposed technique improves the index update throughput by more than an order of magnitude and reduces the performance overhead associated with index updates from 95% to under 15%.

Virtualization provides the possibility of whole machine migration and thus enables a new form of fault tolerance that is completely transparent to applications and operating systems. While initial prototypes show promise, virtualization-based fault-tolerant architecture still experiences substantial performance overhead especially for data-intensive workloads. The main performance challenge of virtualization-based fault tolerance is how to synchronize the memory states of the Master and Slave in a way that minimizes the end-to-end impact on the application performance. This paper describes three optimization techniques for memory state synchronization: fine-grained dirty region identification, speculative state transfer, and synchronization traffic reduction using active slave, and presents a comprehensive performance study of these techniques under three realistic workloads, the TPC-E benchmark, the SPECsfs 2008 CIFS benchmark, and a Microsoft Exchange workload. We show that these three techniques can each reduce the amount of end-of-epoch synchronization traffic by a factor of up to 7, 15 and 5, respectively.

Block-level continuous data protection (CDP) logs every disk block update so that disk updates within a time window are undoable. Standard file servers and DBMS servers can enjoy the data protection service offered by block-level CDP without any modification. Unfortunately, no existing block-level CDP systems can provide users a file versioning view on top of the block versions they maintain. As a result, the data they maintain cannot be used as an extension to the on-line system with which users routinely interact. This paper describes a name-based user-level file versioning system called UVFS that is designed to reconstruct file versions from disk block versions maintained by a block-level CDP. UVFS reconstructs file versions by following the last modified time of files and directories, a common file metadata supported by almost all modern file systems, and therefore does not require any modification to the host file system that a block-level CDP system protects. In addition, UVFS incorporates a file system-specific incremental consistency check mechanism to quickly convert an arbitrary point-in-time block-level snapshot to a file system-consistent one. Performance measurements taken from a fully operational UVFS prototype show that the average end-to-end elapsed time required to discover a file version is under 50 msec from the perspective of an NFS client serviced by an NFS server backed by a block-level CDP system.

In sensor networks we aim to achieve global objectives through local decisions at each node, based only on data available in the node's neighborhood. In this paper, we diffuse information away from source nodes holding desired data, so as to establish information potentials that allow network queries to navigate towards and reach these sources through local greedy decisions, following information gradients. We compute these information potentials by solving for a discrete approximation to a partial differential equation over appropriate network neighborhoods, through a simple local iteration that can be executed in a distributed manner and can be re-invoked to repair the information field locally when links fail, sources move, etc. The solutions to this equation are classical harmonic functions, which have a rich algebraic structure and many useful properties, including the absence of local extrema, providing a guarantee that our local greedy navigation will not get stuck.Unlike shortest path trees, which can also be used to guide queries to sources, information potentials are robust to low-level link volatility as they reflect more global properties of the underlying connectivity. By exploiting the algebraic structure of harmonic functions such potentials can be combined in interesting ways to enable far greater path diversity and thus provide better load balancing than is possible with fixed tree structures, or they can be used to answer range queries about the number of sources in a certain regions by simply traversing the boundary of the region. Potentials for multiple information types can be aggregated and compressed using a variant of the q-digest data structure. The paper provides both analytic results and detailed simulations supporting these claims.

A block-level continuous data protection (CDP) system logs every disk block update from an application server (e.g., a file or DBMS server) to a storage system so that any disk updates within a time window are undoable, and thus is able to provide a more flexible and efficient data protection service than conventional periodic data backup systems. Unfortunately, no existing block-level CDP systems can support arbitrary point-in-time snapshots that are guaranteed to be consistent with respect to the metadata of the application server. This deficiency seriously limits the flexibility in recovery point objective (RTO) of block-level CDP systems from the standpoint of the application servers whose data they protect. This paper describes an incremental file system check mechanism (iFSCK) that is designed to address this deficiency for file servers, and exploits file system-specific knowledge to quickly fix an arbitrary point-in-time block-level snapshot so that it is consistent with respect to file system metadata. Performance measurements taken from a fully operational iFSCK prototype show that iFSCK can turn a 10 GB point-in-time block-level snapshot to be file-system consistent in less than 1 second, and takes less than 25% of the time required by the Fsck utility for vanilla ext3 under relaxed metadata consistency requirements.

As disk storage density increases and data availability requirements become ever more demanding, data replication is increasingly an indispensable feature of enterprise-class storage systems. For highly available storage systems, every disk block is typically replicated on a local mirror server and a remote mirror server in addition to being stored on the main storage server. In a network storage environment, this N-way data replication increases the write traffic load on the storage client's network interface by N times. Multicast is a natural solution to this problem. However, existing storage area network technologies such as fibre channel and Ethernet do not provide adequate support for linklayer multicast. This paper describes a novel reliable multicast mechanism for Ethernet-based storage area networks that effectively exploits the virtual LAN technology and is able to keep the traffic load of N-way replication roughly at the same level as the no-replication case regardless of the value of N. This technology greatly increases the appeal of using Ethernet as the physical-layer technology for storage area networking, as fibre channel networks currently do not support such in-network replication. Performance measurements on an iSCSI-based network storage system demonstrate that the proposed reliable multicast mechanism is able to reduce the end-to-end data transport time by a factor of 2.6 for three-way replication when the disk write size is more than 4 Mbytes.