Post-deduplication delta compression is a data reduction technique that calculates and stores the differences of very similar but non-duplicate chunks in storage systems, which is able to achieve a very high compression ratio. However, the low throughput of widely used resemblance detection approaches (e.g., N-Transform) usually becomes the bottleneck of delta compression systems due to introducing high computational overhead. Generally, this overhead mainly consists of two parts: ① calculating the rolling hash byte by byte across data chunks and ② applying multiple transforms on all of the calculated rolling hash values. In this article, we propose Odess, a fast and lightweight resemblance detection approach, that greatly reduces the computational overhead for resemblance detection while achieving high detection accuracy and a high compression ratio. Odess first utilizes a novel Subwindow-based Parallel Rolling (SWPR) hash method using Single Instruction Multiple Data [ 1 ] (SIMD) to accelerate calculation of rolling hashes (corresponding to the first part of the overhead). Odess then uses a novel Content-Defined Sampling method to generate a much smaller proxy hash set from the whole rolling hash set and quickly applies transforms on this small hash set for resemblance detection (corresponding to the second part of the overhead). Evaluation results show that during the stage of resemblance detection, the Odess approach is ∼31.4× and ∼7.9× faster than the state-of-the-art N-Transform and Finesse (a recent variant of N-Transform [ 39 ]), respectively. When considering an end-to-end data reduction storage system, the Odess-based system’s throughput is about 3.20× and 1.41× higher than the N-Transform- and Finesse-based systems’ throughput, respectively, while maintaining the high compression ratio of N-Transform and achieving ∼1.22× higher compression ratio over Finesse.

Content-Defined Chunking (CDC) has been widely applied in data deduplication systems in the past since it can detect much more redundant data than Fixed-Size Chunking (FSC). CDC approach becomes faster and faster to match the improvement of high performance storage systems, and there are two main kinds of acceleration mechanisms: calculation-efficient acceleration and stream-informed acceleration. We observe the opportunity to combine the benefits of these two mechanisms to chase a faster speed, and the challenges of memory overhead and deduplication ratio loss, which are caused by stream histories and the configuration of minimum/maximum chunk size, respectively. Motivated by these observations, we proposed SuperCDC with several corresponding techniques, including hybridizing calculation-efficient processing with a stream-informed design, memory-efficient structure for tracking stream history, and Min-Max chunking for improving deduplication ratio. Evaluations suggest that SuperCDC achieves an up to 4.94× faster chunking speed and 6.23% higher deduplication ratio while saving 99.58% memory space on stream histories compared with the state-of-the-art Gear-based RapidCDC.