Mixture-of-Experts (MoE) architecture with experts parallelism scales LLMs efficiently by activating only a subset of experts per input, avoiding proportional training costs. However, the intensive and heterogeneous communication substantially hinders the efficiency and scalability of MoE training in the resource-constrained scenario. Existing communication compression techniques fall short in MoE training due to: (i) Intensive training amplifies compression overhead, compromising training efficiency; (ii) Accumulated compression errors propagate through the network, degrading training quality. In this paper, we propose RCMoE, a communication-efficient Random Compression framework for MoE training with two core modules: (1) Local-Stochastic Quantization compresses the all-to-all communication by stochastically quantizing each row of the expert's intermediate computing results in parallel, effectively improving the compression efficiency and reducing compression error; (2) Probabilistic Thresholding Sparsification compresses the all-reduce communication by probabilistically sampling large gradients at high probability, thereby reducing the computational complexity and maintaining the convergence efficiency. Experiments on four typical MoE training tasks prove that RCMoE achieves higher 5.9x-8.1x total communication compression ratios and 1.3x-10.1x training speedup compared with the state-of-the-art compression techniques while maintaining the MoE training accuracy.

HPC applications generate massive amounts of data that impose significant burdens on both storage and I/O systems. Although lossy compressors have been widely adopted in this scenario to reduce data volume, the SOTA approach fails to fully exploit redundancy because it relies on separate techniques that operate at incompatible granularities. Their suboptimal compression ratios leave I/O as the dominant bottleneck in data dumps/loads. Therefore, we propose MAC, a compression framework built upon existing SZ compressor. It leverages the alignment between HPC system characteristics and modeling-based arithmetic coding to balance the compression ratio improvement and time cost. Instead of applying Huffman coding and dictionary-based compressors like zstd or gzip sequentially on quantization factors as SZ does, MAC replaces them with an adaptive arithmetic encoder. Specifically, MAC first employs bit-packing on incoming quantization factors to reduce overhead, as these factors are typically small enough that standard 4-byte storage would impede processing efficiency. The system then constructs context with the knowledge of the length of each quantization factor, utilizing hash tables to store and retrieve historical occurrences. By leveraging two models with distinct prefix-matching strategies and integrating them via a logistic mixer, MAC yields substantial compression gains. This architecture ensures that compression and decompression latencies remain low enough to accelerate overall dump and load operations. Experiments show that MACSZ achieves a compression ratio improvement of over 25%, which translates directly into enhanced throughput on HPC cluster architectures as Fig 1 and 2 demonstrate.

The explosive growth of global data has created an increasing demand for archival storage, where efficient compression is crucial to reduce capacity cost. However, existing archival compressors face a fundamental tradeoff: mainstream methods (e.g., ZSTD with level$21 / 22$) offer limited compression ratios, while context-mixing compressors (e.g., LPAQ) achieve higher ratios but are often too slow for practical use. Therefore, we present CuCM, a GPU-powered context-mixing compressor to overcome this tradeoff. By introducing pre-learning and batch update mechanisms, CuCM resolves the data dependencies inherent in the autoregressive modeling of contextmixing compressors. During compression, CuCM processes each predefined vector as a single unit. It utilizes the current model to predict the probability distribution for the entire vector, deferring model updates until the vector is fully processed. During decompression, CuCM employs an aggressive look-ahead strategy, preassuming bit values for context construction. It then retains only the outcomes of hypotheses that remain consistent with the actual decoded data. Experiments like figure 1 show that CuCM achieves up to$12.6 \times$higher throughput than LPAQ while maintaining comparable compression ratios across both general-purpose and archival datasets.

File systems rely on enforcing storage order for crash recovery. However, this ordering requirement limits I/O parallelism, especially in persistent memory (PM) file systems where ordered I/Os are synchronized for direct access, preventing file systems from fully exploiting PM I/O parallelism.

In backup storage, delta compression successfully achieves a much higher data reduction ratio than chunk-level deduplication by applying a finer granularity in redundancy detection and elimination. However, it introduces additional, intensive computation overhead and results in a 50%–70% worsening backup throughput.

For data reduction techniques used in storage systems, delta compression is often implemented after deduplication, having been shown to achieve a much higher compression ratio by efficiently detecting and compressing similar data chunks. Unfortunately, existing resemblance detection approaches cannot maintain both high throughput and data reduction ratio simultaneously since they either introduce heavy calculation overhead or generate useless features that reduce the accuracy of resemblance detection. In this article, we propose Argus, a fast and precise resemblance detection approach using two primary techniques to improve data-reduction efficiency significantly. First, Argus utilizes a Bin-Wise Partitioning strategy, which separates the rolling hash values of each data chunk into different subsets according to the suffix bits of the hash value and generates features from these subsets. Thus, Argus generates features more efficiently, which both achieves high detection accuracy and improves feature generation speed. Second, based on the efficient Bin-Wise Partitioning strategy, Argus utilizes fine-grained Gear rolling hash and a Plain Feature strategy to manage the granularity of the content represented in the feature, increasing the probability of feature matching and catching as many similar chunks as possible. Consequently, Argus can achieve better detection accuracy, resulting in a much higher compression ratio than previous works while minimizing the computational overhead for resemblance detection. Our evaluation results driven by several real-world datasets suggest that, compared to the state-of-the-art approaches, Argus achieves up to 1.64 × (DeepSketch) and 2.29 × (Finesse, Odess, and N-Transform) higher delta compression ratio and achieves up to 19.9 × (N-Transform), 5.57 × (Finesse), and 1.18 × (Odess) faster feature generation speed.

The rapid growth of container images leads to heavy storage pressures on image registries. Fine-grained deduplication, such as at the file and chunk levels, is a promising technique for reducing the storage space in image registries, compared to Docker's native coarse-grained image layer deduplication. However, fine-grained deduplication often incurs significant image restoration latency due to fragmented I/Os, resulting in up to 8× restoration I/O slowdowns. Moreover, existing restore-optimized deduplication techniques are not tailored to the characteristics of the container image, resulting in low deduplication ratios of images. Consequently, restoring performance remains a critical barrier to the practical adoption of fine-grained deduplication in container image registries. To address this challenge, we propose MiDedup, a restore-friendly fine-grained deduplication approach tailored for container image registries. MiDedup is built on an underexplored observation: the redundancy across layers of images follows a non-uniform distribution. Based on this insight, MiDedup introduces three core techniques: ① Across-layer-aware reorganization, which reorganizes deduplicated data into a compact, sequential layout, significantly reduces fragmented I/Os during image restore. ② Popularity-aware rewriting, which selectively rewrites hot image layers to further improve the balance between deduplication ratio and restore performance. ③ Hybrid-granularity deduplication, which combines file-level and chunk-level deduplication to reduce metadata and fragmented data. Experiments on the real-world Docker image dataset show that MiDedup reduces the 50–80% restore I/O overhead of fine-grained deduplication, which is comparable to the restore performance of native layer-level deduplication, while achieving 1.2–7.3× higher deduplication ratios.

NOR flash has been increasingly popular for RAM-restricted microcontrollers due to its small package, high reliability, etc. To satisfy RAM restrictions, existing NOR flash file systems migrate their functionalities, i.e., block-level data organization and wear leveling (WL), from RAM to NOR flash. However, such fine-grained block-level management introduces frequent index updates and NOR flash scanning, leading to severe I/O amplification, which further deteriorates as they are decoupled in existing NOR flash file systems.

Mixture-of-Experts (MoE) model parallelism is prevalent in training Large Language Models (e.g., ChatGPT). However, the intensive all-to-all collective communication of the MoE layer’s intermediate computing results substantially degrades MoE training efficiency. In this paper, we propose BirdMoE, a novel load-aware communication compression technique with Bi-random quantization for MoE training with two core modules. Specifically, BirdMoE employs a lightweight Random Quantization (RQ) with expectation invariance property to efficiently map the floating-point intermediate computing results into integers while maintaining the MoE training quality. Additionally, BirdMoE utilizes a Mixed Precision (MP) strategy to dynamically balance the communication loads among expert nodes, significantly improving all-to-all communication efficiency for the MoE training system. Experiments on four typical MoE training tasks demonstrate that BirdMoE achieves higher $4.06 \times- 10.44 \times$ total communication compression ratios and $1.18 \times-5.27 \times$ training speedup compared with the state-of-the-art compression techniques while maintaining the MoE training quality.

Current compressors for OLTP databases perform well on text but face challenges with integers, although integers are a critical component of the workload. Most existing integer compressors are ineffective as a complementary solution, since they compress integers together and cannot decompress a certain integer individually, making them incompatible with the data access requirement of OLTP databases. To this end, we propose Apic, a precomputation-based arithmetic coding to efficiently compress each integers (a very tiny unit), though small data are always hard to compress, and ensure compatibility with OLTP datasets. Specifically, Apic presents Bitwidth-aware Precomputed Frequency and Prefixaware Precomputed Decoding to tackle challenges of applying arithmetic coding in this scenario, such as the substantial space costs of symbol frequencies and decompression complexity. Evaluations on real-world and desensitized commercial datasets suggest that Apic improves the compression ratio by up to 80% on integers over VByte, while preserving comparable decompression speed and thus query performance.

Deduplication is widely used in backup storage and reduces storage overhead by allowing backups to share common data chunks. However, it naturally disrupts the sequential layout of backup images, leading to fragmentation, which slows down backup restoration. Existing solutions to this issue often come with trade-offs, either reducing deduplication effectiveness or introducing significant I/O overhead.

Log-structured file system (LFS) has been a popular option for persistent memory (PM) for its high write performance and lightweight crash consistency protocols. However, even with PM's byte-addressable I/O interface, existing PM LFSes still maintain contiguous space for log locality while using heavy garbage collection (GC) to synchronously reclaim space, causing up to 50% I/O performance degradation on PM. Thus, there exists a last-mile problem between the contiguous space management of LFS (that induces GC) and the non-contiguous byte-addressability of PM.

Cryptographically strong hashes are an integral part of deduplicated storage to identify unique chunks, so they are called fingerprints. Many systems were designed with MD5 or SHA-1 hashes, which seemed sufficiently strong at the time. With the ever-increasing computational power and ongoing security attacks, the security and reliability of the original fingerprints in deduplicated storage may be compromised, making them vulnerable to attacks, which can result in data loss or leakage. This may be a concern for deduplicated storage customers that have already purchased systems, who now are required to switch to a stronger hash function such as SHA-256, possibly due to future legal requirements. This reveals the need for upgrading original fingerprints to stronger hashes on existing duplicated storage systems, but there is currently a lack of research on how to perform such an upgrade effectively. Fingerprint upgrade entails not only adding a new fingerprint calculation function but also replacing original fingerprints in existing, large data structures with new fingerprints. One direct method is to first reconstruct all logical files and then deduplicate them again using new fingerprint calculations. However, such a method will repeatedly process duplicate chunks, resulting in significant space and time overhead. In this paper, we propose an upgrade method that decouples computationally and spatially expensive operations from repetitive operations, so that we can release the space of duplicate chunks and avoid excessive space consumption. We also propose leveraging the characteristics of deduplicated data and file similarities to optimize the cache line and data access pattern, respectively, which further reduces the time overhead. Experimental results demonstrate that our upgrade method is practical and efficient, achieving a throughput of$\sim 1.7 \text{GB} / \mathrm{s}$.

Hash indexes are widely used in modern dataintensive applications to support efficient query performance. While persistent memory (PM) provides larger capacity and byteaddressability, we identify that PM-based hash indexes suffer from scalability bottlenecks under high concurrency. This stems from NUMA access imposing significant costs, exacerbating the inherently request-agnostic I/O issues in concurrent read/write operations. To this end, we present ALPHA, a highly scalable hash index designed for persistent memory. The key idea with ALPHA lies in exploiting the natural hash-based data access features of hash indexes to reduce I/O irrelevant to data requests. We propose a NUMA-friendly framework to reduce the negative impacts of remote access through the cross-thread delegation mechanism. To further improve throughput, we use two techniques to address read and write issues of the PM-oriented hash index. First, we propose a novel two-layer hash structure to minimize PM access by filtering unnecessary key retrievals during data reads. Second, we devise a partition-based finegrained data access mechanism that enables lock-free concurrent writes without excessive PM write overheads. Our comprehensive experimental evaluation shows that ALPHA outperforms state-of-the-art PM-based hash indexes by up to$22.17 \times$in throughput while reducing tail latency by up to 97.1 %.

The row-oriented store model is the cornerstone component of modern online transaction processing (OLTP) database systems. In response to the massive increase in data within database systems, compression techniques are employed to enhance storage efficiency. Regrettably, current compression methods suffer from either the amplification issue due to coarse compression granularity or inefficient decompression operations, thus usually decreasing the speed of query processing. To this end, we present DPTC, a cost-effective and decompression-transparent approach designed to compress data pages, the basic storage unit of OLTP database systems. Specifically, (1) DPTC applies a row-wise decompression-oriented structure to track the first occurrence of redundant data in compressed data, which effectively supports the decompression of individual records from pages, thereby avoiding unwarranted decompression in record access. Moreover, (2) DPTC employs an in-page dynamic packing strategy, which determines the compression units based on the impact of each data reduction operation on the compression gains and eliminates gains-inefficient data reductions. Furthermore, (3) DPTC utilizes a SIMD-based mechanism that leverages the characteristics of operations within the decompression process to improve the decompression speed. Our evaluation results confirm that DPTC is efficient in terms of decompression speed and compression ratio. Within an OLTP database system, DPTC yields throughput improvements of up to 4.28 x in TPC-C and reduces latency by up to 33.3% for data point queries in a row-oriented storage engine.

UCEs (Uncorrectable memory errors) pose significant challenges to cloud computing systems, often resulting in catastrophic failures and crashes. Researchers have explored prediction approaches to address this issue. Previous studies have provided insights into memory error prediction, focusing on memory module part numbers and relationships between error code data. However, these efforts face challenges due to insufficient data features and suboptimal optimization, especially in production environments where hardware/software sparing techniques are widely deployed, the UCE ratio is low, and long lead time is required. To address these issues, our study first collect a large amount of memory data from different vendors in Huawei's production environment, which has deployed hardware/software sparing techniques, to provide more general data. Second, we exploit new indicators termed UPH (Unique, Pinx, and History) from this data, which play a crucial role in predicting UCEs. UPH offers a more profound understanding of the factors contributing to UCEs and demonstrates higher precision and recall. Then, we integrate existing indicators and UPH into our prediction framework and demonstrate the significance of UPH through indicator importance assessments. We also optimize the framework by determining an optimal sampling window. In production environments with long lead time and low UCE ratio, we improve the framework by implementing noise reduction, self-history learning, and a new scenario-based model selection approach. Experimental results demonstrate 19 % - 27 % increase in UCE prediction recall with 4 %-11 % increase in precision under different scenarios, outperforming state-of-the-art methods in production environments.

The rapid growth of Docker images in cloud infrastructures has intensified storage and network demands, posing challenges to QoS for registries and deployments. This paper systematically evaluates four data reduction methods—file-level and block-level deduplication, delta compression, and local compression—on 18 representative images. We quantify their tradeoffs in compression, computation, I/O overhead, and restore performance, revealing that (1) filtering large files ($>64 \text{KB}$) preserves 80% of redundancy elimination at half the cost, (2) category-based reorganization reduces restore latency by 83%, and (3) fixed-size chunking with optimized blocks balances memory and compression. Based on these insights, we propose adaptive strategies for efficient container storage.

A dynamic perfect hashing index offers collision-free assignment for entries, ensuring excellent query performance, an essential metric for the Quality of Service (QoS) in database and big data systems. Its core mechanism is rehashing, which resolves collisions during insertions by reassigning entries to distinct slots. However, rehashing is costly, as hash collisions increase linearly with the number of entries in the hash bucket, and the exponential complexity of finding a feasible reassignment. Therefore, rehashing significantly limits the insertion performance of dynamic perfect hashing indexes. To this end, we propose PDPH, a hybrid approach that enhances insertion performance by combining static and dynamic perfect hashing techniques. Specifically, (1) PDPH leverages the complementary strengths of both techniques to reduce the rehashing complexity to a linear level, rather than the exponential overhead of previous approaches. (2) It further reduces the frequency of rehashing by enforcing multiple insertions to share a single rehashing operation. (3) It also limits the scale of rehashing by allowing partial rehashing operations. Evaluations on realworld datasets suggest that, compared to the state-of-the-art approach (MapEmbed), PDPH enhances insertion throughput by up to$2.32 \times$, while maintaining query performance.

Given two graphs H 1 and H 2 , a graph is ( H 1 , H 2 ) -free if it contains no induced subgraph isomorphic to H 1 or H 2 . A graph G is k -vertex-critical if every proper induced subgraph of G has chromatic number less than k , but G has chromatic number k . The study of k -vertex-critical graphs for specific graph classes is an important topic in algorithmic graph theory because if the number of such graphs that are in a given hereditary graph class is finite, then there exists a polynomial-time certifying algorithm to decide the k -colorability of a graph in the class. In this paper, we show that: (1) for k ≥ 1 , there are finitely many k -vertex-critical ( P 5 , K 1 , 4 + P 1 ) -free graphs; (2) for s ≥ 1 , there are finitely many 5-vertex-critical ( P 5 , K 1 , s + P 1 ) -free graphs; (3) for k ≥ 1 , there are finitely many k -vertex-critical ( P 5 , K 3 + 2 P 1 ‾ ) -free graphs. Moreover, we characterize all 5-vertex-critical ( P 5 , H ) -free graphs where H ∈ { K 1 , 3 + P 1 , K 1 , 4 + P 1 , K 3 + 2 P 1 ‾ } using an exhaustive graph generation algorithm.

Fine-grained deduplication (also known as delta compression) can achieve a better deduplication ratio compared to chunk-level deduplication. This technique removes not only identical chunks but also reduces redundancies between similar but non-identical chunks. Nevertheless, it introduces considerable I/O overhead in deduplication and restore processes, hindering the performance of these two processes and rendering fine-grained deduplication less popular than chunk-level deduplication to date. In this paper, we explore various issues that lead to additional I/O overhead and tackle them using several techniques. Moreover, we introduce MeGA, which attains fine-grained deduplication/restore speed nearly equivalent to chunk-level deduplication while maintaining the significant deduplication ratio benefit of fine-grained deduplication. Specifically, MeGA employs (1) a backup-workflow-oriented delta selector and cache-centric resemblance detection to mitigate poor spatial/temporal locality in the deduplication process, and (2) a delta-friendly data layout and “Always-Forward-Reference” traversal to address poor spatial/temporal locality in the restore workflow. Evaluations on four datasets show that MeGA achieves a better performance than other fine-grained deduplication approaches. Specifically, MeGA significantly outperforms the traditional greedy approach, providing 10–46 times better backup speed and 30–105 times more efficient restore speed, all while preserving a high deduplication ratio.

QLC SSDs have gained increasing popularity in cloud computing, PCs, and smartphones due to their low prices and high density, but they suffer from extremely limited endurance. Deduplication can convert redundant chunk writes into fine-grained metadata updates, thereby promising to alleviate QLC wear. Nevertheless, our observation shows that existing deduplication approaches cause even more I/Os than non-deduplication systems. We find that the amplification comes from two sources: (1) I/O amplification due to the mismatched granularity between SSD I/O size (e.g., 4--16 KiB page) and deduplication metadata I/O size, and (2) SSD garbage collection (GC) amplification due to deduplication metadata updates for eliminating redundant chunks (i.e., increment reference count). To address the above problem, this paper proposes H2C-Dedup, which employs two essential techniques. First, to address I/O amplification, cold2hot-heating technique utilizes a log-structured metadata I/O scheme, which ensures that the deduplication metadata is flushed until it is accumulated within I/O cache to match with the OS I/O granularity. Second, to address GC amplification, hot2cold-suppression technique divides metadata into hot (e.g., to store reference count) and cold (e.g., to store fingerprint) segments, delta-encoding the hot entries to ensure both hot and cold metadata will not be modified once they are durable. As a result, H2C-Dedup significantly reduces the deduplication-induced I/O and GC amplification. We implement H2C-Dedup based on F2FS. Extensive experiments on the FEMU platform using microbenchmarks and real-world traces indicate that H2C-Dedup can extend to at most 3.3× and 3.4× lifespan while accelerating 26% and 37% I/O performance compared to SmartDedup and HF-Dedupe.

As time series data become popular, their volume increases rapidly. Time series databases are designed for such data, and they process data in short slices, meaning that the compression units for compressors are small. How to compress the short slices of floating-points while reserving a high compression ratio and a high decompression speed remains a problem.To solve the problem, we propose a lossy compressor Machete. It uses an efficient hybrid encoder of Huffman encoding and variable length quantity (VLQ). Adaptive encoding selection makes it excel on short-slice data compression ratio, while the simple framework ensures fast decompression. We also find a limitation in VLQ and propose the optimal VLQ to further improve the compression ratio.Our evaluation on four real-world datasets shows that Machete outperforms state-of-the-art compressors by 32%−80% on compression ratio and achieves the fastest decompression speed on two datasets. When applied to a well-known time series database InfluxDB, Machete saves disk usage up to 79% and improves the query performance of the InfluxDB database by saving I/O.

Deduplication-based techniques are popular in backup storage systems for reducing data volume. To maximize data reduction, existing fine-grained deduplication approaches not only eliminate duplicate chunks but also delta-compress non-duplicate chunks as delta relative to their similar (base) chunks. However, each chunk may have multiple similar chunks, and delta compression usually only selects one of them as the base chunk, i.e., a one-to-one scheme. This scheme benefits the restore performance because it needs to read only one (instead of multiple) base chunk in decompressing delta chunks, while it also wastes the potential compressibility among other similar chunks.In this paper, we propose SuperDelta to further exploit compressibility across multiple similar chunks and to preserve the restore performance advantage of the one-to-one scheme as much as possible. It is based on three techniques. (1) To further eliminate redundancy among similar chunks, SuperDelta applies a "Multiple Referenced Base Chunks" (MRBC) scheme instead of the one-to-one scheme. It combines several similar pairs of chunks in delta encoding to recover possibly lost compressibility in "boundary shift" problems. (2) To avoid the negative side effects of MRBC on restore performance, SuperDelta introduces a rebase scheme to rebuild simple reference paths among duplicate and similar chunks. It significantly simplifies the restore workflow, but also costs slightly more storage space because of impacting the workflow of redundancy detection. (3) To compensate for the additional storage cost, SuperDelta applies a space-recycle scheme to remove derived data when they become old while ensuring the optimized restore performance of the latest backups.Experiments on four real-world backup datasets show that SuperDelta increases the overall compression ratio by 1.05~2.40 times than the traditional one-to-one fine-grained deduplication without significantly affecting the backup and restore throughput.

The adoption of disaggregated memory (DM) in key-value (KV) storage systems is considered a cost-effective and efficient solution for addressing the significant performance challenges encountered by conventional KV storage systems. However, these systems must handle substantial concurrent requests, making it essential to detect and resolve conflicts to ensure the data correctness. Existing approaches guarantee the correctness of concurrent operations by Compare And Swap (CAS) but consume more network round-trip times (RTTs) to degrade performance. In addition, previous methods incur additional overhead when DM nodes fail.To address the above issues, this paper introduces AKV, a high-performance Agent-Based Key-Value Store on disaggregated memory. AKV partitions keys according to specified strategies, where a single partition’s keys are managed by the same agent to handle read and write requests from multiple clients. This design mitigates the likelihood of concurrency conflicts by enforcing fine-grained serialization of requests within each partition. Specifi-cally, to partition keys, AKV proposes load-aware and affinity-aware strategies. To handle concurrent requests in a fine-grained serialized manner, AKV introduces a partition-level concurrency control scheme without RDMA_CAS. To detect the agent failure and recovery for high availability, AKV proposes a decentralized approach without additional management servers. We evaluate AKV with micro and real-world benchmarks. Experimental results show that AKV outperforms the state-of-the-art KV stores on DM by up to 1.8 × in throughput.

NFS, which is commonly used in local area networks for sharing files and data, is frequently employed in setting up internal network storage systems for enterprises. In these scenarios, performance and security are the main concerns for users. NFS-Ganesha is the user-space version of NFS that offers a lot of extensions and powerful features. However, compared to kernel-based NFS, it has some performance drawbacks and more security implications.To address these issues, the paper proposes kNFS, which builds on NFS-Ganesha by integrating eBPF for performance optimization and security enhancement. kNFS offers a higher-performing and more secure network file system by using caching for quick replies to repeated requests and achieving faster handling of read requests in the kernel. It also uses a more efficient algorithm for packet filtering, providing more fine-grained ACL rules and network traffic monitoring to enhance security. The experimental results indicate that kNFS significantly improves performance for various operations, with almost no impact on unoptimized operation handling. In terms of resource overhead, it requires only a small amount of additional memory and does not demand extra CPU resources. In addition, compared to IpTables, kNFS’s ACL rule checking is faster, especially when there are a large number of rules.

Storage Area Networks (SANs) are critical in enterprise IT infrastructure, offering high-throughput and low-latency storage solutions. However, traditional SAN architectures that rely on kernel-space multipathing software face significant challenges in virtualized environments, where extended IO paths and high context-switching overheads degrade performance. While the Storage Performance Development Kit (SPDK) has emerged as a powerful tool to enhance IO efficiency by enabling userspace drivers, it introduces limitations in multipathing capabilities.To address these challenges, we have developed SUMF (SPDK Userspace IO Multipathing Framework), an innovative userspace IO multipathing software. SUMF leverages SPDK’s strengths while overcoming its limitations by directly managing multiple IO paths within virtualized cloud computing environments. This solution reduces IO latency by avoiding unnecessary user-kernel transitions and optimizes the system through improved IO path aggregation and selection, dynamic load balancing, and robust failover and failback mechanisms. Our experiments demonstrate that, compared to native SPDK, SUMF achieves higher throughput and shows stronger adaptability to path failures and bandwidth imbalances during transmission.

In order to achieve high availability and low storage costs in distributed storage systems, erasure code is widely used instead of replication. Compared to replication, erasure code can reduce storage costs, but also brings higher repair costs. There are currently many repair algorithms to reduce the block reconstruction time of single block failure. However, applying the existing methods to multi-block failures may lead to unbalanced network traffic, unnecessary network transfers, and network congestion at data collection node during the repair process, which can not make full use of the bandwidth between nodes.To solve this problem, we propose a novel repair algorithm called Partial Merge Repair (PMR) for multi-block failures, which is a scheduling algorithm that considers network load between nodes and combines multiple failed blocks to recover together. It first divides all surviving nodes into different groups, and then the data collection nodes within the group collect the data needed to repair multiple blocks through cross merging. Finally, the data collection node sends the collected blocks to the repair node to complete the repair. Our study presents a formal definition and proof of network transfer time in the modeled repair process of PMR, highlighting its superior efficiency compared to existing methods in homogeneous environments.We implement a prototype of PMR to evaluate its performance. The experimental results indicate that compared to existing repair technologies, PMR improves repair throughput by 28%-256% for various scenes.

With the increasing scale of data and user demands for low latency, the development of large-scale clusters has become a trend. To ensure high availability of data in data clusters, XOR-based erasure code fault-tolerant technologies are widely used due to their low storage and computational overhead. Meanwhile, as the scale of clusters ranges from hundreds to thousands, the probability of multiple node failures is not negligible. This can lead to serious consequences, such as data loss, and should be recovered as soon as possible. However, codes such as RDP and EVENODD can easily lead to network congestion when recovering in the event of concurrent failures, making it challenging to recover quickly.To address this issue, we propose a novel network transfer plan search algorithm, Greedy Row-Diagonal Parity Search or GRS for short. GRS optimally allocates the network traffic generated during the repair process by greedily utilizing idle bandwidth and leveraging the commutative property of XOR operations, ensuring a more even distribution of traffic across the cluster network, which improves the repair throughput.We build a prototype in a distributed erasure-coded cluster and conduct experiment evaluation. The experimental results indicate that, compared to existing repair optimization methods, GRS improves repair throughput by 230%-880%.

Thread scheduling is a core functionality of operating systems, optimizing CPU resource utilization through dedicated algorithms. However, modifying kernel-level scheduling in Linux is complex and error-prone. User-space scheduling offers a promising alternative, yet existing implementations struggle with speed, flexibility, and robustness. This paper introduces F4, a user-space thread scheduling framework that shifts scheduling algorithm deployment from the kernel to user space. By allowing the kernel to delegate scheduling decisions through interfaces like system calls and eBPF hooks, F4 enables user space to develop and implement its own scheduling strategies. This separation of scheduling mechanisms and policy definitions enhances efficiency and adaptability. F4 improves communication through message queues and the shoot system, accelerates execution with inter-processor interrupts (IPI), and increases interference resistance through dynamic priority adjustments. These innovations allow F4 to achieve fast, flexible, and stable-fortified thread scheduling in user space, addressing the limitations of current user-space scheduling solutions.

Image data storage has grown explosively, so image deduplication is used to save storage by eliminating redundancy between different images. However, traditional image deduplication cannot eliminate fine-grained redundancy nor guarantee lossless results. In this work, we propose imDedup, a lossless and fine-grained deduplication scheme for JPEG image storage systems. Specifically, imDedup uses a novel sampling hash method, Feature Bitmap, to detect similar images in a fast way by utilizing the information distribution of JPEG data. Meanwhile, it uses Idelta, a novel delta encoder that incorporates image compression into deduplication, to guarantee the non-redundant data can be re-compressed via image encoding and thus improves the compression ratio. Besides, we propose the DCHash and Fixed-Point Matching (FPM) techniques to further speed up Idelta. We also propose imDedup-plus, which dynamically chooses the DCHash-based or FPM-based compressor to achieve higher throughputs without sacrificing the compression ratio. Experimental results demonstrate the superiority of the imDedup-based methods on five datasets. Compared with the state-of-the-art similarity detector and delta encoder, imDedup achieves 1.8–4.4× higher throughputs and 1.3–1.7× higher compression ratios, respectively. Besides, imDedup-plus can further achieve 1.3–2.9× higher throughputs than imDedup without sacrificing the compression ratio.

Persistent Memory (PM), with its low latency, byte-addressability, and non-volatility, sparks various PM file systems. Nevertheless, we observe that these file systems can waste more than 50% PM I/O bandwidth, hindering the value of PM. Our in-depth analysis reveals that this can be rooted in their strong consistency assumptions that require synchronized PM I/O for strict data integrity. Nevertheless, these I/Os, including attribute updates, space management, and journaling, are generally small, random, and fail to match with the PM I/O characteristics. We present HUNTER, a POSIX-compliant PM file system that aims to unleash PM I/O performance with strict data integrity. Our key insight is delaying crash consistency with emerging flush-on-fail (FoF) hardware. We achieve this insight by first leveraging a PM-simplified soft update mechanism to persist metadata (i.e., consistent view) in the background; while maintaining up-to-date states in DRAM (i.e., latest view) for instantaneous user responding. Compared to prior works, HUNTER carefully decouples the latest view from the consistent view, and thus, background operations will not block the foreground paths, which fully exploits soft update efficiency. Under this architecture, FoF hardware allows a residual energy window for state synchronization during system failures. However, ensuring strict data integrity is still non-trivial due to existing deficient sync mechanisms that might either hide the benefits of soft update or fail to flush all states during the FoF window. We propose collaborated synchronization that bridges the two views to minimize PM I/O to address the issue. Our extensive experiments show that HUNTER can achieve 1.35–7.51× I/O throughput compared to existing PM file systems. Under the strict workload configurations, HUNTER outperforms the fastest PM file system by 72–122%.

Top-K sparsification-based compression techniques are popular and powerful for reducing communication costs in federated learning (FL). However, existing Top-K sparsification-based compression methods suffer from two critical issues that severely hinder their implementation, particularly in the context of FL, which often involves a vast number of resource-constrained devices: 1) the low compressibility of the Top-K parameter’s indexes significantly limits the overall compression ratio (CR) and 2) the residual accumulation techniques used to maintain the model quality consume huge memory resources. To address these issues, we propose a novel FL compression framework, named FedComp, for deep neural networks (DNNs). FedComp achieves a higher communication CR while maintaining comparable model quality at low memory cost. Specifically, FedComp incorporates the following three key components: 1) a tensor-wise index-sharing mechanism that greatly reduces the index proportion by sharing one index among multiple elements of the tensor; 2) a fine-grained parameters packing strategy that reduces the transmission of duplicate value and index by considering their properties, thereby further reducing the overall communication cost; and 3) a residual compressor that significantly reduces memory cost by enhancing the compressibility of floating-point residuals and achieving a high CR with a lossless encoding scheme. Experiments on mainstream machine learning (ML) tasks with different DNN structures and datasets demonstrate that our proposed FedComp outperforms the state-of-the-art FL compression algorithms by achieving a higher communication CR of up to$28.5\times $while reducing memory costs by$21.04\times $–$50.59\times $on the local residual model, without degrading FL training performance.

Delta encoding is a data reduction technique capable of calculating the differences (i.e., delta) among very similar files and chunks. It is widely used for various applications, such as synchronization replication, backup/archival storage, cache compression, and so on. However, delta encoding is computationally costly due to its time-consuming word-matching operations for delta calculation. Existing delta encoding approaches either run at a slow encoding speed, such as Xdelta and Zdelta, or at a low compression ratio, such as Ddelta and Edelta. In this article, we propose Gdelta, a fast delta encoding approach with a high compression ratio. The key idea behind Gdelta is the combined use of five techniques: (1) employing an improved Gear-based rolling hash to replace Adler32 hash for fast scanning overlapping words of similar chunks, (2) adopting a quick array-based indexing for word-matching, (3) applying a sampling indexing scheme to reduce the cost of traditional building full indexes for base chunks’ words, (4) skipping unmatched words to accelerate delta encoding through non-redundant areas, and (5) last but not least, after word-matching, further batch compressing the remainder to improve the compression ratio. Our evaluation results driven by seven real-world datasets suggest that Gdelta achieves encoding/decoding speedups of 3.5X∼25X over the classic Xdelta and Zdelta approaches while increasing the compression ratio by about 10%∼240%.

The Top-K sparsification-based compression framework is extensively explored for reducing communication costs in distributed learning. However, we identified several issues with existing Top-K sparsification-based compression methods: (i) The limited compressibility of the Top-K parameter's indexes critically restricts the overall communication compression ratio; (ii) Several time-consuming compression operations significantly offset the benefits of communication compression; (iii) The use of error feedback techniques to maintain model quality results in a high memory footprint consumption. To solve these issues, we propose BIRD, a lightweight tensor-wiseBi-Random samplingstrategy with an expectation invariance property. Specifically, BIRD applies a tensor-wiseindex sharingmechanism that reduces the index proportion by allowing multiple tensor elements to share a single index, thus improving the overall compression ratio. Additionally, BIRD replaces the time-consuming Top-K sorting with a fasterBi-Random samplingstrategy based on the aforementionedindex sharingmechanism, significantly reducing compression overheads; Moreover, BIRD establishes anexpectation invarianceproperty into theBi-Random samplingto ensure an approximate unbiased representation for the$L_1$-norm of the sampled tensors, effectively maintaining the model quality without incurring extra memory costs. We further optimize BIRD to BIRD+ by introducing the uniform distribution-based sampling and Gamma correction on the tensor-wise sampling process, achieving a more flexibly adjustment of the sparsity with better convergence performance. Experimental evaluations across multiple conventional distributed learning tasks demonstrate that compared to state-of-the-art approaches, BIRD+ achieves higher communication compression ratios up to 36.2$\times$and higher computation throughput up to 149.6$\times$while maintaining the model quality without incurring extra memory costs.

Persistent transactional memory (PTM) exploits transactions to provide an easy crash-consistent interface for persistent memory (PM). However, because of the substantial reader-side overhead brought on by the low bandwidth and long persistence latency of PM, present PTM research cannot scale effectively. This paper proposes a highly scalable PTM system, DRPTM, which allows nearly non-overhead reads without lowering the isolation level. DRPTM decouples persistence latency from concurrency control and traces the read-only copy maintained in logs as a lightweight read set. The evaluation shows that DRPTM significantly outperforms the state-of-the-art PTM systems for various workloads and achieves near-linear scalability.

We present HUNTER, a POSIX-compliant persistent memory (PM) file system that fully releases PM’s write performance. Compared to state-of-the-art ones, HUNTER proposes a novel PM-DRAM collaboration architecture to significantly eliminate/reduce software overheads in the write path. Expensive in-PM metadata are updated asynchronously to hide their performance penalties. Furthermore, in-PM metadata/data are laid out separately for locality awareness, enabling collaboration with asynchronous architecture. HUNTER also adopts several lightweight in-DRAM allocators/indexes to manage PM efficiently.Experimental results suggest that HUNTER achieves 2.0–3.4× write bandwidth compared to state-of-the-art PM file systems in write-intensive workloads and shows similar write bandwidth compared to bare PM.

Top-K sparsification-based compression framework is widely employed to reduce communication costs in distributed learning. However, we have identified several issues with existing Top-K sparsification-based compression methods that severely impede their deployment in resource-constrained devices: (i) the limited compressibility of the Top-K parameter’s indexes, which critically restricts the overall communication compression ratio; (ii) several time-consuming compression operations significantly negate the benefits of communication compression; (iii) the high memory footprint consumption associated with error feedback techniques used to maintain model quality.To address these issues, we propose a lightweight tensor-wise Bi-Random sampling strategy with expectation invariance property called BIRD, which achieves higher compression ratios at lower computational overheads while maintaining a comparable model quality without additional memory costs. Specifically, BIRD applies a tensor-wise index sharing mechanism that substantially reduces the proportion of the index by allowing multiple tensor elements to share a single index, thus improving the overall compression ratio. Additionally, BIRD replaces the time-consuming Top-K sorting with a faster Bi-Random sampling strategy based on the aforementioned index sharing mechanism, thereby reducing the computational costs of compression; Moreover, BIRD establishes an expectation invariance property into the above Bi-Random sampling to ensure an unbiased representation for the L1-norm of the sampled tensors, effectively maintaining the model quality without incurring extra memory costs.Experiments on multiple mainstream machine learning (ML) tasks demonstrate that compared to state-of-the-art methods, our proposed BIRD achieves 1.3×-31.1× higher compression ratio at lower time overheads with O(N) complexity while maintaining the model quality without incurring extra memory costs.

In recent years, the performance of hash indexes has been significantly improved by exploiting emerging persistent memory (PMem). However, the performance improvement of hash indexes mainly comes from exploiting the hardware features of PMem. Only a few studies optimize the hash index itself to fully exploit the potential of PMem. Interestingly, many of these studies improve the performance of write, but disregard the performance of read, of hash indexes on PMem. With extensive experimental evaluation, we find the major reason for inefficient read in the hash index on PMem is that the overhead of hash collision processing is expensive.To address that, we propose a novel Efficient Extendible Perfect Hashing (EEPH) on PMem-DRAM hybrid data layout to improve read performance of hash indexes. Specifically, we reduce the overhead of dynamic perfect hashing extension on PMem by combing extendible hashing. We then design a hybrid data layout to unlock the inherent read strengths of perfect hashing (i.e., zero collision). Last, we devise a complement move algorithm to efficiently guarantee the zero collision of perfect hashing when data move is conducted on PMem. We compare EEPH with the state-of-the-art hash indexes on PMem by conducting comprehensive experiments on several real-world read-intensive and read-skew workloads. The experimental results confirm the superiority of our EEPH as it achieves up to 2.21× higher throughput and about 1/3 of the 99th percentile latency than state-of-the-art hash indexes.

Maintaining the high availability of online systems requires reliable and fast online anomaly detection and fault localization. However, existing anomaly detection methods either suffer high training costs and low generalization capabilities or are designed and evaluated using offline data with limited efficacy in online usage. Furthermore, these methods' fault localization capabilities are often inadequate due to external observability constraints. Therefore, designing a new approach to address these limitations effectively is essential. To address the aforementioned limitations, this paper proposes a novel non-intrusive in situ anomaly detection and fault lo-calization framework, Detective-Dee. The proposed framework leverages a compressed sensing method for anomaly detection, which exhibits strong generalization capabilities and eliminates extensive training. Detective-Dee further improves its performance by incorporating three optimization techniques: concurrent sub-stitution sampling, Look-Up-Table-based similarity calculation, and substitution window-based threshold selection to improve parallelism and reduce computational and comparison overheads. Additionally, the framework adopts an innovative non-intrusive fault localization strategy based on anomaly detection triggering. This approach utilizes the dynamic instrumentation capabilities of eBPF, combined with extracting vulnerable function and function call chains through source code analysis, to improve the online anomaly detection capability and achieve robust fault localization with low overhead. To validate the effectiveness of Detective-Dee, we developed a prototype system and conducted a comprehensive evaluation. The results demonstrate that, compared to the state-of-the-art anomaly detection method, Detective-Dee exhibits a 4x improve-ment in anomaly detection speed while maintaining higher online and comparable offline detection ability. Furthermore, under 33 real-world fault cases across eight popular distributed systems, Detective-Dee successfully detects 31 cases and accurately locates 26 cases with less than 1% overhead, outperforming the state-of-the-art method.

Erasure codes are widely deployed in modern storage systems, leading to frequent usage of their encoding/decoding operations. The encoding/decoding process for erasure codes is generally carried out using the parity-check matrix approach. However, this approach is serial and computationally expensive, mainly due to dealing with matrix operations, which results in low encoding/decoding performance. These drawbacks are particularly evident for newer erasure codes, including SD and LRC codes. To address these limitations, this article introduces the Partitioned and Parallel Matrix ( PPM ) algorithm. This algorithm partitions the parity-check matrix, parallelizes encoding/decoding operations, and optimizes calculation sequence to facilitate fast encoding/decoding of these codes. Furthermore, we present a generalized PPM ( gPPM ) algorithm that surpasses PPM in performance by employing fine-grained dynamic matrix calculation sequence selection. Unlike PPM, gPPM is also applicable to erasure codes such as RS code. Experimental results demonstrate that PPM improves the encoding/decoding speed of SD and LRC codes by up to 210.81%. Besides, gPPM achieves up to 102.41% improvement over PPM and 32.25% improvement over RS regarding encoding/decoding speed.

Deep Neural Networks (DNNs) have achieved remarkable success in various real-world applications. However, running a Deep Neural Network (DNN) typically requires hundreds of megabytes of memory footprints, making it challenging to deploy on resource-constrained platforms such as mobile devices and IoT. Although mainstream DNNs compression techniques such as pruning, distillation, and quantization can reduce the memory overhead of model parameters during DNN inference, they suffer from three limitations: (i) low model compression ratio for the lightweight DNN structures with little redundancy, (ii) potential degradation in model inference accuracy, and (iii) inadequate memory compression ratio is attributable to ignoring the layering property of DNN inference. To address these issues, we propose a lightweight memory-efficient DNN inference framework called Smart-DNN+, which significantly reduces the memory costs of DNN inference without degrading the model quality. Specifically, ① Smart-DNN+ applies a layerwise binary-quantizer with a remapping mechanism to greatly reduce the model size by quantizing the typical floating-point DNN weights of 32-bit to the 1-bit signs layer by layer. To maintain model quality, ② Smart-DNN+ employs a bucket-encoder to keep the compressed quantization error by encoding the multiple similar floating-point residuals into the same integer bucket IDs. When running the compressed DNN in the user’s device, ③ Smart-DNN+ utilizes a partially decompressing strategy to greatly reduce the required memory overhead by first loading the compressed DNNs in memory and then dynamically decompressing the required materials for model inference layer by layer. Experimental results on popular DNNs and datasets demonstrate that Smart-DNN+ achieves lower 0.17%–0.92% memory costs at lower runtime overheads compared with the states of the art without degrading the inference accuracy. Moreover, Smart-DNN+ potentially reduces the inference runtime up to 2.04× that of conventional DNN inference workflow.

Post-deduplication delta compression is a data reduction technique that calculates and stores the differences of very similar but non-duplicate chunks in storage systems, which is able to achieve a very high compression ratio. However, the low throughput of widely used resemblance detection approaches (e.g., N-Transform) usually becomes the bottleneck of delta compression systems due to introducing high computational overhead. Generally, this overhead mainly consists of two parts: ① calculating the rolling hash byte by byte across data chunks and ② applying multiple transforms on all of the calculated rolling hash values. In this article, we propose Odess, a fast and lightweight resemblance detection approach, that greatly reduces the computational overhead for resemblance detection while achieving high detection accuracy and a high compression ratio. Odess first utilizes a novel Subwindow-based Parallel Rolling (SWPR) hash method using Single Instruction Multiple Data [ 1 ] (SIMD) to accelerate calculation of rolling hashes (corresponding to the first part of the overhead). Odess then uses a novel Content-Defined Sampling method to generate a much smaller proxy hash set from the whole rolling hash set and quickly applies transforms on this small hash set for resemblance detection (corresponding to the second part of the overhead). Evaluation results show that during the stage of resemblance detection, the Odess approach is ∼31.4× and ∼7.9× faster than the state-of-the-art N-Transform and Finesse (a recent variant of N-Transform [ 39 ]), respectively. When considering an end-to-end data reduction storage system, the Odess-based system’s throughput is about 3.20× and 1.41× higher than the N-Transform- and Finesse-based systems’ throughput, respectively, while maintaining the high compression ratio of N-Transform and achieving ∼1.22× higher compression ratio over Finesse.

Deep neural networks (DNNs) have achieved remarkable success in many fields. However, large-scale DNNs also bring storage costs when storing snapshots for preventing clusters’ frequent failures or incur significant communication overheads when transmitting DNNs in the Federated Learning (FL). Recently, several approaches, such as Delta-DNN and LC-Checkpoint, aim to reduce the size of DNNs’ snapshot storage by compressing the difference between two neighboring versions of the DNNs (a.k.a., delta). However, we observe that existing approaches, applying traditional global lossy quantization techniques in DNN's delta compression, can not fully exploit the data similarity since the parameters’ value ranges vary among layers. To fully explore the similarity of the delta model and improve the compression ratio, we propose a quantization-based local-sensitive delta compression approach, named QD-Compressor, by developing a layer-based local-sensitive quantization scheme and error feedback mechanism. Specifically, the quantizers and number of quantization bits are adaptive among layers based on the value distribution and weighted entropy of the delta's parameters. To avoid quantization error degrading the performance of the restored model, an alternative error feedback mechanism is designed to dynamically correct the quantization error during the training process. Experiments on multiple popular DNNs and datasets show that QD-Compressor obtains a higher 7×-40× compression ratio in the model snapshot compression scenario than the state-of-the-art approaches. Additionally, QD-Compressor achieves an 11×-15× compression ratio to the residual model of the Federated Learning compression scenario.

Top-k sparsification method is popular and powerful forreducing the communication cost in Federated Learning(FL). However, according to our experimental observation, it spends most of the total communication cost on the index of the selected parameters (i.e., their position informa-tion), which is inefficient for FL training. To solve this problem, we propose a FL compression algorithm for convolution neural networks (CNNs), called SmartIdx, by extending the traditional Top-k largest variation selection strategy intothe convolution-kernel-based selection, to reduce the proportion of the index in the overall communication cost and thusachieve a high compression ratio. The basic idea of SmartIdx is to improve the 1:1 proportion relationship betweenthe value and index of the parameters to n:1, by regarding the convolution kernel as the basic selecting unit in parameter selection, which can potentially deliver more informationto the parameter server under the limited network traffic. Tothis end, a set of rules are designed for judging which kernel should be selected and the corresponding packaging strategies are also proposed for further improving the compressionratio. Experiments on mainstream CNNs and datasets show that our proposed SmartIdx performs 2.5×−69.2× higher compression ratio than the state-of-the-art FL compression algorithms without degrading model performance.

Content-Defined Chunking (CDC) has been widely applied in data deduplication systems in the past since it can detect much more redundant data than Fixed-Size Chunking (FSC). CDC approach becomes faster and faster to match the improvement of high performance storage systems, and there are two main kinds of acceleration mechanisms: calculation-efficient acceleration and stream-informed acceleration. We observe the opportunity to combine the benefits of these two mechanisms to chase a faster speed, and the challenges of memory overhead and deduplication ratio loss, which are caused by stream histories and the configuration of minimum/maximum chunk size, respectively. Motivated by these observations, we proposed SuperCDC with several corresponding techniques, including hybridizing calculation-efficient processing with a stream-informed design, memory-efficient structure for tracking stream history, and Min-Max chunking for improving deduplication ratio. Evaluations suggest that SuperCDC achieves an up to 4.94× faster chunking speed and 6.23% higher deduplication ratio while saving 99.58% memory space on stream histories compared with the state-of-the-art Gear-based RapidCDC.

Since one chunk could be shared by many files after data deduplication, Garbage Collection (GC) is an essential but complex task to reclaim stale chunks in large-scale primary deduplication systems. Traditional Mark&Sweep is a widely used approach but suffers from the increasingly traversing time and huge memory overhead of Liveness Array (i.e., a data structure reflects the liveness of alive chunks) in the Mark phase. This paper proposes a new method named Focused Garbage Collection (FGC) to accelerate the Mark phase for primary deduplication storage significantly. Specifically, we design a global Austere Reference Graph with low memory cost that efficiently represents files’ reference relationships (i.e., sharing chunks after deduplication) by considering the deduplication characteristics of workloads in primary systems. Austere Reference Graph helps FGC focus on the deleted files and their correlative files to quickly mark stale chunks, while traditional approaches need to traverse all files. Consequently, FGC’s traversing time and Liveness Array size will be greatly reduced in the Mark phase. Evaluation results show that compared with traditional Mark&Sweep, FGC decreases the time consumption in the Mark phase 1.3×-7.34× in a stand-alone primary deduplication system and 128×-256× network traffic reduction for the Mark phase while only introducing < 0.05% extra memory overhead for the reference graph.

Images occupy a large amount of storage in data centers. To cope with the explosive growth of the image storage requirement, image compression techniques are devised to shrink the size of every single image at first. Furthermore, image deduplication methods are proposed to reduce the storage cost as they could be used to eliminate redundancy among images. However, state-of-the-art image deduplication methods either can only eliminate file-level coarse-grained redundancy or cannot guarantee lossless deduplication. In this work, we propose a new lossless image deduplication framework to eliminate fine-grained redundancy among images. It first decodes images to expose similarity, then eliminates fine-grained redundancy on the decoded data by delta compres-sion, and finally re-compresses the remaining data by image compression encoding. Based on this framework, we propose a novel lossless similarity-based deduplication (SBD) scheme for decoded image data (called imDedup). Specifically, imDedup uses a novel and fast sampling method (called Feature Map) to detect similar images in a two-dimensional way, which greatly reduces computation overhead. Meanwhile, it uses a novel delta encoder (called Idelta) which incorporates image compression encoding characteristics into deduplication to guarantee the remaining deduplicated image data to be friendly re-compressed via image encoding, which significantly improves the compression ratio. We implement a prototype of imDedup for JPEG images, and demonstrate its superiority on four datasets: Compared with exact image deduplication, imDedup achieves a 19%-38% higher compression ratio by efficiently eliminating fine-grained redundancy. Compared with the similarity detector and delta encoder of state-of-the-art SBD schemes running on the decoded image data, imDedup achieves a 1.8×-3.4× higher throughput and a 1.3 ×-1. 6 × higher compression ratio, respectively.

Content-Defined Chunking(CDC) is the key stage of data deduplication since it has a significant impact on deduplication system’s throughput and deduplication efficiency. However, existing CDC algorithms suffer from high computation overhead, weak stability, and poor ability to handle low-entropy strings. In this paper, we propose UltraCDC, a fast and stable, high-efficiency deal with low-entropy strings, CDC algorithm for deduplication-based storage systems. There are four key techniques behind UltraCDC, namely, rolling compute boundary conditions, skipping sub-minimum chunk size, normalized chunking, and jumping to detect low-entropy strings. Using a sliding window to rolling compute boundary conditions not only accelerates the chunking stage but also makes it more resistant to boundary shift, the two techniques of skipping sub-minimum chunk size and normalized chunking can complement each other to speed up chunking without sacrificing deduplication ratio too much, and the jumping detection can detect more low-entropy strings than AE-opt2 without affecting chunking speed. We implemented UltraCDC in Destor, and the experimental results show that using the above four techniques, chunking speed is 1.5–10× faster than the state-of-the-art CDC approaches, while deduplication ratio is comparable or even higher than the classic Rabin-base CDC. In terms of the capability to detect low-entropy strings, UltraCDC is a CDC approach with the highest ability to detect low-entropy strings, 102× and 2× higher than Rabin-based CDC and AE-opt2, respectively.

Fake news detection attracts many researchers’ attention due to the negative impacts on the society. Most existing fake news detection approaches mainly focus on semantic analysis of news’ contents. However, the detection performance will dramatically decrease when the content of news is short. In this paper, we propose a novelfake news detection multi-task learning (FDML)model based on the following observations: 1) some certain topics have higher percentages of fake news; and 2) some certain news authors have higher intentions to publish fake news. FDML model investigates the impact of topic labels for the fake news and introduce contextual information of news at the same time to boost the detection performance on the short fake news. Specifically, the FDML model consists of representation learning and multi-task learning parts to train the fake news detection task and the news topic classification task, simultaneously. As far as we know, this is the first fake news detection work that integrates the above two tasks. The experiment results show that the FDML model outperforms state-of-the-art methods on real-world fake news dataset.

Data deduplication is widely used to reduce the size of backup workloads, but it has the known disadvantage of causing poor data locality, also referred to as the fragmentation problem. This results from the gap between the hyper-dimensional structure of deduplicated data and the sequential nature of many storage devices, and this leads to poor restore and garbage collection (GC) performance. Current research has considered writing duplicates to maintain locality (e.g., rewriting) or caching data in memory or SSD, but fragmentation continues to lower restore and GC performance. Investigating the locality issue, we design a method to flatten the hyper-dimensional structured deduplicated data to a one-dimensional format, which is based on classification of each chunk’s lifecycle, and this creates our proposed data layout. Furthermore, we present a novel management-friendly deduplication framework, called MFDedup, that applies our data layout and maintains locality as much as possible. Specifically, we use two key techniques in MFDedup: Neighbor-duplicate-focus indexing (NDF) and Across-version-aware Reorganization scheme (AVAR). NDF performs duplicate detection against a previous backup, then AVAR rearranges chunks with an offline and iterative algorithm into a compact, sequential layout, which nearly eliminates random I/O during file restores after deduplication. Evaluation results with five backup datasets demonstrate that, compared with state-of-the-art techniques, MFDedup achieves deduplication ratios that are 1.12× to 2.19× higher and restore throughputs that are 1.92× to 10.02× faster due to the improved data layout. While the rearranging stage introduces overheads, it is more than offset by a nearly-zero overhead GC process. Moreover, the NDF index only requires indices for two backup versions, while the traditional index grows with the number of versions retained.

Delta sync (synchronization) is a key bandwidth-saving technique for cloud storage services. The representative delta sync utility,rsync, matches data chunks by sliding a search window byte-by-byte to maximize the redundancy detection for bandwidth efficiency. However, it is difficult for this process to cater to the forthcoming high-bandwidth cloud storage services which require lightweight delta sync that can well support large files. Moreover,rsyncemploys invariant chunking and compression methods during the sync process, making it unable to cater to services from various network environments which require the sync approach to perform well under different network conditions. Inspired by the Content-Defined Chunking (CDC) technique used in data deduplication, we propose NetSync, a network adaptive and CDC-based lightweight delta sync approach with less computing and protocol (metadata) overheads than the state-of-the-art delta sync approaches. Besides, NetSync can choose appropriate compressing and chunking strategies for different network conditions. The key idea of NetSync is (1) to simplify the process of chunk matching by proposing a fast weak hash called FastFP that is piggybacked on the rolling hashes from CDC, and redesigning the delta sync protocol by exploiting deduplication locality and weak/strong hash properties; (2) to minimize the sync time by adaptively choosing chunking parameters and compression methods according to the current network conditions. Our evaluation results driven by both benchmark and real-world datasets suggest NetSync performs$2\times$2×–$10\times$10×faster and supports$30\%$30%–$80\%$80%more clients than the state-of-the-artrsync-based WebR2sync+ and deduplication-based approach.

Deep neural networks (DNNs) have gained considerable attention in various real-world applications due to their strong performance in representation learning. However, running a DNN needs tremendous memory resources, which significantly restricts DNN from being applicable on resource-constrained platforms (e.g., IoT, mobile devices, etc.). Lightweight DNNs can accommodate the characteristics of mobile devices, but the hardware resources of mobile or IoT devices are extremely limited, and the resource consumption of lightweight models needs to be further reduced. However, the current neural network compression approaches (i.e., pruning, quantization, knowledge distillation, etc.) works poorly on the lightweight DNNs, which are already simplified. In this paper, we present a novel framework called Smart-DNN, which can efficiently reduce the memory requirements of running DNNs on resource-constrained platforms. Specifically, we slice a neural network into several segments and use SZ error-bounded lossy compression to compress each segment separately while keeping the network structure unchanged. When running a network, we first store the compressed network into memory and then partially decompress the corresponding part layer by layer. According to experimental results on four popular lightweight DNNs (usually used in resource-constrained platforms), Smart-DNN achieves memory saving of 1/10∼1/5, while slightly sacrificing inference accuracy and unchanging the neural network structure with accepted extra runtime overhead.

Deep neural networks (DNNs) have achieved remarkable success in many fields. Large-scale DNNs also bring storage challenges when storing snapshots for preventing clusters’ frequent failures, and bring massive internet traffic when dispatching or updating DNNs for resource-constrained devices (e.g., IoT devices, mobile phones). Several approaches are aiming to compress DNNs. The Recent work, Delta-DNN, notices high similarity existed in DNNs and thus calculates differences between them for improving the compression ratio.However, we observe that Delta-DNN, applying traditional global lossy quantization technique in calculating differences of two neighboring versions of the DNNs, can not fully exploit the data similarity between them for delta compression. This is because the parameters’ value ranges (and also the delta data in Delta-DNN) are varying among layers in DNNs, which inspires us to propose a local-sensitive quantization scheme: the quantizers are adaptive to parameters’ local value ranges in layers. Moreover, instead of quantizing differences of DNNs in Delta-DNN, our approach quantizes DNNs before calculating differences to make the differences more compressible. Besides, we also propose an error feedback mechanism to reduce DNNs’ accuracy loss caused by the lossy quantization.Therefore, we design a novel quantization-based delta compressor called QD-Compressor, which calculates the lossy differences between epochs of DNNs for saving storage cost of backing up DNNs’ snapshots and internet traffic of dispatching DNNs for resource-constrained devices. Experiments on several popular DNNs and datasets show that QD-Compressor obtains a compression ratio of 2.4× ~ 31.5× higher than the state-of-the-art approaches while well maintaining the model’s test accuracy.

Multiple data reduction techniques have been investigated to lower storage costs for a wide variety of customers. In this work, we focus on similarity-based delta compression, which calculates and stores the difference of very similar, but non-duplicate, chunks in storage systems. Delta compression is often implemented along with deduplication and has been shown to achieve a much higher compression ratio. Currently, the N-Transform method is the most popular and widely-used approach to generate features for data content (e.g. chunks) to detect similar candidates (and then apply delta compression). For delta compression systems, though, the throughput of N-Transform is often the bottleneck. Finesse is a high throughput variant of N-Transform, but it suffers from lower detection accuracy and compression ratio. The computation overhead of N-Transform consists of two parts: calculating the rolling hash across data and applying time-consuming transforms on each hash. In this work, we propose Odess, a fast resemblance detection approach, that uses a novel Content-Defined Sampling method to generate a much smaller proxy hash set and then applies transforms on this small hash set. This reduces the calculations in the transform step from being the bottleneck. Meanwhile, Odess also leverages the faster Gear hash to generate rolling hashes. Thus, Odess greatly reduces the computational overhead for resemblance detection while achieving high detection accuracy and high compression ratio. Our evaluation results show that Odess is ~ 5.4× (Finesse) and ~ 26.9× (N-Transform) faster (on average) at generating features for resemblance detection. When considering an end-to-end data reduction storage system, Odess increases throughput by ~ 1.36× (Finesse) and ~ 2.76× (N-Transform) while maintaining the compression ratio of N-Transform and increasing the compression ratio ~ 1.22× over Finesse.

Delta compression (or called delta encoding) is a data reduction technique capable of calculating the differences (i.e., delta) among the very similar files and chunks, and is thus widely used for optimizing synchronization replication, backup/archival storage, cache compression, etc. However, delta compression is costly because of its time-consuming word-matching operations for delta calculation. Existing delta encoding approaches, are either at a slow encoding speed, such as Xdelta and Zdelta, or at a low compression ratio, such as Ddelta and Edelta. In this paper, we propose Gdelta, a fast delta encoding approach with a high compression ratio, that improves the delta encoding speed by employing an improved fast Gear-based rolling hash for scanning fine-grained words, and a quick array-based indexing scheme for word-matching, and then, after word-matching, further batch compressing the rest to improve the compression ratio. Our evaluation results driven by six real-world datasets suggest that Gdelta achieves encoding/decoding speedups of 2X~4X over the classic Xdelta and Zdelta approaches while increasing the compression ratio by about 10%~120%.

Deep neural networks (DNNs) have gained considerable attention in various real-world applications due to the strong performance on representation learning. However, a DNN needs to be trained many epochs for pursuing a higher inference accuracy, which requires storing sequential versions of DNNs and releasing the updated versions to users. As a result, large amounts of storage and network resources are required, significantly hampering DNN utilization on resource-constrained platforms (e.g., IoT, mobile phone).

Content-Defined Chunking (CDC) has been playing a key role in data deduplication systems recently due to its high redundancy detection ability. However, existing CDC-based approaches introduce heavy CPU overhead because they declare the chunk cut-points by computing and judging the rolling hashes of the data stream byte by byte. In this article, we propose FastCDC, a Fast and efficient Content-Defined Chunking approach, for data deduplication-based storage systems. The key idea behind FastCDC is the combined use of five key techniques, namely, gear based fast rolling hash, simplifying and enhancing the Gear hash judgment, skipping sub-minimum chunk cut-points, normalizing the chunk-size distribution in a small specified region to address the problem of the decreased deduplication ratio stemming from the cut-point skipping, and last but not least, rolling two bytes each time to further speed up CDC. Our evaluation results show that, by using a combination of the five techniques, FastCDC is 3-12X faster than the state-of-the-art CDC approaches, while achieving nearly the same and even higher deduplication ratio as the classic Rabin-based CDC. In addition, our study on the deduplication throughput of FastCDC-based Destor (an open source deduplication project) indicates that FastCDC helps achieve 1.2-3.0X higher throughput than Destor based on state-of-the-art chunkers.

Scientific simulations in high-performance computing (HPC) environments generate vast volume of data, which may cause a severe I/O bottleneck at runtime and a huge burden on storage space for postanalysis. Unlike traditional data reduction schemes such as deduplication or lossless compression, not only can error-controlled lossy compression significantly reduce the data size but it also holds the promise to satisfy user demand on error control. Pointwise relative error bounds (i.e., compression errors depends on the data values) are widely used by many scientific applications with lossy compression since error control can adapt to the error bound in the dataset automatically. Pointwise relative-error-bounded compression is complicated and time consuming. We develop efficient precomputation-based mechanisms based on the SZ lossy compression framework. Our mechanisms can avoid costly logarithmic transformation and identify quantization factor values via a fast table lookup, greatly accelerating the relative-error-bounded compression with excellent compression ratios. In addition, we reduce traversing operations for Huffman decoding, significantly accelerating the decompression process in SZ. Experiments with eight well-known real-world scientific simulation datasets show that our solution can improve the compression and decompression rates (i.e., the speed) by about 40 and 80 p, respectively, in most of cases, making our designed lossy compression strategy the best-in-class solution in most cases.

Data deduplication, as a proven technology for effective data reduction in backup and archive storage systems, also demonstrates the promise in increasing the logical space capacity of storage caches by removing redundant data. However, our in-depth evaluation of the existing deduplication-aware caching algorithms reveals that they do improve the hit ratios compared to the caching algorithms without deduplication, especially when the cache block size is set to 4KB. But when the block size is larger than 4KB, a clear trend for modern storage systems, their hit ratios are significantly reduced. A slight increase in hit ratios due to deduplicationmay not be able to improve the overall storage performance because of the high overhead created by deduplication. To address this problem, in this paper we propose CDAC, a Content-driven Deduplication-Aware Cache, which focuses on exploiting the blocks' content redundancy and their intensity of content sharing among source addresses in cache management strategies. We have implemented CDAC based on LRU and ARC algorithms, called CDAC-LRU and CDAC-ARC respectively. Our extensive experimental results show that CDACLRU and CDAC-ARC outperform the state-of-the-art deduplication-aware caching algorithms, D-LRU and DARC, by up to 19.49X in read cache hit ratio, with an average of 1.95X under real-world traces when the cache size ranges from 20% to 80% of the working set size and the block size ranges from 4KB to 64 KB.

Scientific simulations in high-performance computing (HPC) environments are producing vast volume of data, which may cause a severe I/O bottleneck at runtime and a huge burden on storage space for post-analysis. Unlike the traditional data reduction schemes (such as deduplication or lossless compression), not only can error-controlled lossy compression significantly reduce the data size but it can also hold the promise to satisfy user demand on error control. Point-wise relative error bounds (i.e., compression errors depends on the data values) are widely used by many scientific applications in the lossy compression, since error control can adapt to the precision in the dataset automatically. Point-wise relative error bounded compression is complicated and time consuming. In this work, we develop efficient precomputation-based mechanisms in the SZ lossy compression framework. Our mechanisms can avoid costly logarithmic transformation and identify quantization factor values via a fast table lookup, greatly accelerating the relative-error bounded compression with excellent compression ratios. In addition, our mechanisms also help reduce traversing operations for Huffman decoding, and thus significantly accelerate the decompression process in SZ. Experiments with four well-known real-world scientific simulation datasets show that our solution can improve the compression rate by about 30% and decompression rate by about 70% in most of cases, making our designed lossy compression strategy the best choice in class in most cases.

In order to save human lives and reduce injury and property loss, Situation Awareness (SA) information is essential and important for rescue workers to perform the effective and timely disaster relief. The information is generally derived from the shared images via widely used smartphones. However, conventional smartphone-based image sharing schemes fail to efficiently meet the needs of SA applications due to two main reasons, i.e., real-time transmission requirement and application-level image redundancy, which is exacerbated by limited bandwidth and energy availability. In order to provide efficient image sharing in disasters, we propose a bandwidth- and energy- efficient image sharing system, called BEES. The salient feature behind BEES is to propose the concept of Approximate Image Sharing (AIS), which explores and exploits approximate feature extraction, redundancy detection, and image uploading to trade the slightly low quality of computation results in content-based redundancy elimination for higher bandwidth and energy efficiency. Nevertheless, the boundaries of the tradeoffs between the quality of computation results and efficiency are generally subjective and qualitative. We hence propose the energy-aware adaptive schemes in AIS to leverage the physical energy availability to objectively and quantitatively determine the tradeoffs between the quality of computation results and efficiency. Moreover, unlike existing work only for cross-batch similar images, BEES further eliminates in-batch ones via a similarity-aware submodular maximization model. The response time of querying similar images is reduced via leveraging a geographic coordinate based index partitioning scheme. We have implemented the BEES prototype which is evaluated via three real-world image datasets. Extensive experimental results demonstrate the efficacy and efficiency of BEES. We have released the source codes of BEES at GitHub.

Data deduplication is able to effectively identify and eliminate redundant data and only maintain a single copy of files and chunks. Hence, it is widely used in distributed storage systems and cloud storage to save the users' network bandwidth for uploading files. However, the occurrence of deduplication can be easily identified by monitoring and analyzing network traffic, which leads to the risk of user privacy leakage. An attacker can carry out a very dangerous side channel attack, i.e., learn-the-remaining-information (LRI) attack, to reveal users' privacy information by exploiting the side channel of network traffic in deduplication. Existing work addresses the LRI attack at the cost of the high bandwidth consumption. In order to address this problem, we propose a simple yet effective scheme, called randomized redundant chunk scheme (RRCS), to significantly mitigate the risk of the LRI attack while maintaining the high bandwidth efficiency of deduplication. The idea behind RRCS is to add randomized redundant chunks to mix up the real deduplication states of files used for the LRI attack, which effectively obfuscates the view of the attacker, who attempts to exploit the side channel of network traffic for the LRI attack. Our security analysis shows that RRCS significantly mitigates the risk of the LRI attack. We have implemented the RRCS prototype and evaluated it by using three real-world datasets. Experimental results demonstrate RRCS significantly outperforms existing work in terms of bandwidth efficiency.

As promising alternatives in building future main memory systems, emerging non-volatile memory (NVM) technologies can increase memory capacity in a cost-effective and power-efficient way. However, NVM is facing security threats due to its limited write endurance: a malicious adversary can wear out the cells and cause the NVM system to fail quickly. To address this issue, several wear-leveling schemes have been proposed to evenly distribute write traffic in a security-aware manner. In this study, we present a new type of timing attack, remapping timing attack (RTA), based on information leakage from the remapping latency difference in NVM. Our analysis and experimental results show that RTA can cause three of the latest wear-leveling schemes (i.e., region-based start-gap, security refresh, and multi-way wear leveling) to lose their effectiveness in several days (even minutes), causing failure of NVM. To defend against such an attack, we further propose a novel wear-leveling scheme called the ‘security region-based start-gap (security RBSG)’, which is a two-stage strategy using a dynamic Feistel network to enhance the simple start-gap wear leveling with level-adjustable security assurance. The theoretical analysis and evaluation results show that the proposed security RBSG not only performs well when facing traditional malicious attacks, but also better defends against RTA.

Data deduplication [3] is able to effectively identify and eliminate redundant data and only maintain a single copy of files and chunks. Hence, it is widely used in cloud storage systems to save the users' network bandwidth for uploading data. However, the occurrence of deduplication can be easily identified by monitoring and analyzing network traffic, which leads to the risk of user privacy leakage. The attacker can carry out a very dangerous side channel attack, i.e., learn-the-remaining-information (LRI) attack, to reveal users' privacy information by exploiting the side channel of network traffic in deduplication [1]. In the LRI attack, the attacker knows a large part of the target file in the cloud and tries to learn the remaining unknown parts via uploading all possible versions of the file's content. For example, the attacker knows all the contents of the target file X except the sensitive information θ. To learn the sensitive information, the attacker needs to upload m files with all possible values of θ, respectively. If a file Xd with the value θd is deduplicated and other files are not, the attacker knows that the information θ = θd.

In order to save human lives and reduce injury and property loss, Situation Awareness (SA) information is essential and important for rescue workers to perform the effective and timely disaster relief. The information is generally derived from the shared images via widely used smartphones. However, conventional smartphone-based image sharing schemes fail to efficiently meet the needs of SA applications due to two main reasons, i.e., real-time transmission requirement and application-level image redundancy, which is exacerbated by limited bandwidth and energy availability. In order to provide efficient image sharing in disasters, we propose a bandwidth-and energy-efficient image sharing system, called BEES. The salient feature behind BEES is to propose the concept of Approximate Image Sharing (AIS), which explores and exploits approximate feature extraction, redundancy detection, and image uploading to trade the slightly low quality of computation results in content-based redundancy elimination for higher bandwidth and energy efficiency. Nevertheless, the boundaries of the tradeoffs between the quality of computation results and efficiency are generally subjective and qualitative. We hence propose the energy-aware adaptive schemes in AIS to leverage the physical energy availability to objectively and quantitatively determine the tradeoffs between the quality of computation results and efficiency. Moreover, unlike existing work only for cross-batch similar images, BEES further eliminates in-batch ones via a similarity-aware submodular maximization model. We have implemented the BEES prototype which is evaluated via three real-world image datasets. Extensive experimental results demonstrate the efficacy and efficiency of BEES.

Live migration of virtual machines (VM) is one of the key characteristics of virtualization for load balancing, system maintenance, power management, etc., in data centers or clusters. In order to reduce the data transferred and shorten the migration time, the compression techniques have been widely used to accelerate VM migration. However, different compression approaches have different compression ratios and speeds. Because there is a trade-off between compression and transmission, the migration performance improvements obtained from different compression approaches are differentiated, and the improvements vary with the network bandwidth. Besides, the compression window sizes used in most compression algorithms are typically much larger than a single page size, so the traditional single page compression loses some potential compression benefits. In this paper, we design and implement a Bandwidth-Aware Compression (BAC) scheme for VM migration. BAC chooses suitable compression approach according to the network bandwidth available for the migration process, and employs multi-page compression. These features make BAC obtain more migration performance improvements from compression. Experiments under various network scenarios demonstrate that, compared with conventional compression approaches, BAC shortens the total migration time while achieving comparable performance for the total data transferred and the downtime.

Chunk-level deduplication, while robust in removing duplicate chunks, introduces chunk fragmentation which decreases restore performance. Rewriting algorithms are proposed to reduce the chunk fragmentation and accelerate the restore speed. Delta compression can remove redundant data between non-duplicate but similar chunks which cannot be eliminated by chunk-level deduplication. Some applications use delta compression as a complement for chunk-level deduplication to attain extra space and bandwidth savings. However, we observe that delta compression introduces a new type of chunk fragmentation stemming from delta compressed chunks whose base chunks are fragmented. We refer to such delta compressed chunks as base-fragmented chunks. We found that this new type of chunk fragmentation has a more severely impact on the restore performance than the chunk fragmentation introduced by chunk-level deduplication and cannot be reduced by existing rewriting algorithms. In order to address the problem due to the base-fragmented chunks, we propose SDC, a scheme that selectively performs delta compression after chunk-level deduplication. The main idea behind SDC is to simulate a restore cache to identify the non-base-fragmented chunks and only perform delta compression for these chunks, thus avoiding the new type of chunk fragmentation. Due to the locality among the backup streams, most of the non-base-fragmented chunks can be detected by the simulated restore cache. Experimental results based on real-world datasets show that SDC improves the restore performance of the delta compressed and deduplicated backup system by 1.93X-7.48X, and achieves 95.5%-97.4% of its compression, while imposing negligible impact on the backup throughput.

Chunk-level deduplication plays an important role in backup storage systems. Existing Content-Defined Chunking (CDC) algorithms, while robust in finding suitable chunk boundaries, face the key challenges of (1) low chunking throughput that renders the chunking stage a serious deduplication performance bottleneck, (2) large chunk size variance that decreases deduplication efficiency, and (3) being unable to find proper chunk boundaries in low-entropy strings and thus failing to deduplicate these strings. To address these challenges, this paper proposes a new CDC algorithm called the Asymmetric Extremum (AE) algorithm. The main idea behind AE is based on the observation that the extreme value in an asymmetric local range is not likely to be replaced by a new extreme value in dealing with the boundaries-shifting problem. As a result, AE has higher chunking throughput, smaller chunk size variance than the existing CDC algorithms, and is able to find proper chunk boundaries in low-entropy strings. The experimental results based on realworld datasets show that AE improves the throughput performance of the state-of-the-art CDC algorithms by more than 2.3×, which is fast enough to remove the chunking-throughput performance bottleneck of deduplication, and accelerates the system throughput by more than 50 percent, while achieving comparable deduplication efficiency.

Data compression is widely used in storage systems to reduce redundant data and thus save storage space. One challenge facing the traditional compression approaches is the limitation of compression windows size, which fails to reduce redundancy globally. In this paper, we present DEC, a Deduplication-Enhanced Compression approach that effectively combines deduplication and traditional compressors to increase compression ratio and efficiency. Specifically, we make full use of deduplication to (1) accelerate data reduction by fast but global deduplication and (2) exploit data locality to compress similar chunks by clustering the data chunks which are adjacent to the same duplicate chunks. Our experimental results of a DEC prototype based on real-world datasets show that DEC increases the compression ratio by 20% to 71% and speeds up the compression throughput by 17%~183% compared to traditional compressors, without sacrificing the decompression throughput by leveraging deduplication in traditional compression approaches.

Tencent, one of the biggest Internet companies in China, contains billions of users and over 600-PB data, and leverages thousands of SSDs in the storage system to improve system performance and obtain energy savings. Existing commercial SSDs however fail to meet the needs of the ultra largescale applications due to not matching the service patterns. In order to address this problem and deliver high performance, we propose an application-aware and software-defined SSD scheme for Tencent applications, called TSSD. TSSD explores and exploits the business characteristics of Tencent, which facilitates the efficient use of SSDs. TSSD is software-defined by packaging each flash chip as a fully independent and concurrent storage unit. Each concurrent unit can be mounted as a character device, which allows the application layer to manage the flash chips in a more efficient manner, while optimizing the data layout. TSSD further employs a host-target FTL (TFTL) that uses a dedicated interface in the application layer, which efficiently connects the application layer with flash chips. Application layer hence becomes more accurately by using the flash memory chip-level information from TFTL, including the storage utilization, the degree of wear, etc. Moreover, TFTL is a programmable FTL and provides a programmable interface to the application layer. According to the running states of SSDs and workload information, TSSD makes use of the programmable interface to efficiently improve the performance of the FTL, wear leveling, and garbage collection for the specified applications. Extensive experiments use the real-world datasets from the commercial storage systems of Tencent. The results demonstrate that TSSD significantly improves the storage system performance and meets the needs of the Tencent's large-scale business applications.

As an emerging memory technology to build the future main memory systems, Phase Change Memory (PCM) can increase memory capacity in a cost-effective and power-efficient way. However, PCM is facing security threats for its limited write endurance: a malicious adversary could wear out the cells and cause the whole PCM system to fail within a short period of time. To address this issue, several wear-leveling schemes have been proposed to evenly distribute write traffic in a security-aware manner. In this work, we present a new type of timing attacknamed Remapping Timing Attack (RTA), based on the asymmetry in write time of PCM. Our analysis and experimental results show that the new revealed RTA can make two state-of-the-art wear-leveling schemes (Region Based Start-Gap and Security Refresh) lose effectiveness, failing PCM with these two techniques in several days (even minutes). In order to defend such attack, we further propose a novel wear-leveling scheme called Security Region Based Start-Gap (Security RBSG), which employs a two-stage strategy and uses a dynamic Feistel Network to enhance the simple Start-Gap wear leveling with level-adjustable security assurance. The theoretical analysis and evaluation results show that the proposed Security RBSG is the most robust wear-leveling methodology so far, which not only better defends the new RTA, but also performs well on the traditional malicious attacks, i.e., Repeated Address Attack and Birthday Paradox Attack.

NAND flash memory has been widely used as storage medium in diverse environments due to its light weight, low power consumption, and high I/O performance. Page replacement is an important operation in NAND flash- based storage systems. However, traditional replacement algorithms designed for magnetic disks fail to meet the needs of NAND flash memory due to inherent features such as asymmetric I/O latencies and erase-before- write. In order to address this problem, this paper proposes a new page replacement algorithm, called Ghost buffer Assisted and Self-tuning Algorithm (GASA). GASA reduces expensive flash write operations by evicting cold clean pages preferentially and maintains reasonable buffer hit ratios via a ghost buffer. In addition, GASA is self-tuning due to the use of a simple learning scheme, thus adaptively matching different workloads and buffer sizes. Experimental results based on real-world OLTP traces demonstrate that GASA offers a good trade-off between the hit ratio and the flash write count, thus achieving better I/O performance than the state-of-the-art page replacement algorithms.

Data deduplication, an efficient approach to data reduction, has gained increasing attention and popularity in large-scale storage systems due to the explosive growth of digital data. It eliminates redundant data at the file or subfile level and identifies duplicate content by its cryptographically secure hash signature (i.e., collision-resistant fingerprint), which is shown to be much more computationally efficient than the traditional compression approaches in large-scale storage systems. In this paper, we first review the background and key features of data deduplication, then summarize and classify the state-of-the-art research in data deduplication according to the key workflow of the data deduplication process. The summary and taxonomy of the state of the art on deduplication help identify and understand the most important design considerations for data deduplication systems. In addition, we discuss the main applications and industry trend of data deduplication, and provide a list of the publicly available sources for deduplication research and studies. Finally, we outline the open problems and future research directions facing deduplication-based storage systems.

Data reduction has become increasingly important in storage systems due to the explosive growth of digital data in the world that has ushered in the big data era. One of the main challenges facing large-scale data reduction is how to maximally detect and eliminate redundancy at very low overheads. In this paper, we present DARE, a low-overhead deduplication-aware resemblance detection and elimination scheme that effectively exploits existing duplicate-adjacency information for highly efficient resemblance detection in data deduplication based backup/archiving storage systems. The main idea behind DARE is to employ a scheme, call Duplicate-Adjacency based Resemblance Detection (DupAdj), by considering any two data chunks to be similar (i.e., candidates for delta compression) if their respective adjacent data chunks are duplicate in a deduplication system, and then further enhance the resemblance detection efficiency by an improved super-feature approach. Our experimental results based on real-world and synthetic backup datasets show that DARE only consumes about 1/4 and 1/2 respectively of the computation and indexing overheads required by the traditional super-feature approaches while detecting 2-10 percent more redundancy and achieving a higher throughput, by exploiting existing duplicate-adjacency information for resemblance detection and finding the “sweet spot” for the super-feature approach.

In backup systems, the chunks of each backup are physically scattered after deduplication, which causes a challenging fragmentation problem. We observe that the fragmentation comes into sparse and out-of-order containers. The sparse container decreases restore performance and garbage collection efficiency, while the out-of-order container decreases restore performance if the restore cache is small. In order to reduce the fragmentation, we propose History-Aware Rewriting algorithm (HAR) and Cache-Aware Filter (CAF). HAR exploits historical information in backup systems to accurately identify and reduce sparse containers, and CAF exploits restore cache knowledge to identify the out-of-order containers that hurt restore performance. CAF efficiently complements HAR in datasets where out-of-order containers are dominant. To reduce the metadata overhead of the garbage collection, we further propose a Container-Marker Algorithm (CMA) to identify valid containers instead of valid chunks. Our extensive experimental results from real-world datasets show HAR significantly improves the restore performance by 2.84-175.36 × at a cost of only rewriting 0.5-2.03 percent data.

Data deduplication, a space-efficient and bandwidth-saving technology, plays an important role in bandwidth-efficient data transmission in various data-intensive network and cloud applications. Rabin-based and MAXP-based Content-Defined Chunking (CDC) algorithms, while robust in finding suitable cut-points for chunk-level redundancy elimination, face the key challenges of (1) low chunking throughput that renders the chunking stage the deduplication performance bottleneck and (2) large chunk-size variance that decreases deduplication efficiency. To address these challenges, this paper proposes a new CDC algorithm called the Asymmetric Extremum (AE) algorithm. The main idea behind AE is based on the observation that the extreme value in an asymmetric local range is not likely to be replaced by a new extreme value in dealing with the boundaries-shift problem, which motivates AE's use of asymmetric (rather than symmetric as in MAXP) local range to identify cut-points and simultaneously achieve high chunking throughput and low chunk-size variance. As a result, AE simultaneously addresses the problems of low chunking throughput in MAXP and Rabin and high chunk-size variance in Rabin. The experimental results based on four real-world datasets show that AE improves the throughput performance of the state-of-the-art CDC algorithms by 3x while attaining comparable or higher deduplication efficiency.

Nowadays, many customers and enterprises backup their data to cloud storage that performs deduplication to save storage space and network bandwidth. Hence, how to perform secure deduplication becomes a critical challenge for cloud storage. According to our analysis, the state-of-the-art secure deduplication methods are not suitable for cross-user finegrained data deduplication. They either suffer brute-force attacks that can recover files falling into a known set, or incur large computation (time) overheads. Moreover, existing approaches of convergent key management incur large space overheads because of the huge number of chunks shared among users. Our observation that cross-user redundant data are mainly from the duplicate files, motivates us to propose an efficient secure deduplication scheme SecDep. SecDep employs User-Aware Convergent Encryption (UACE) and Multi-Level Key management (MLK) approaches. (1) UACE combines cross-user file-level and inside-user chunk-level deduplication, and exploits different secure policies among and inside users to minimize the computation overheads. Specifically, both of file-level and chunk-level deduplication use variants of Convergent Encryption (CE) to resist brute-force attacks. The major difference is that the file-level CE keys are generated by using a server-aided method to ensure security of cross-user deduplication, while the chunk-level keys are generated by using a user-aided method with lower computation overheads. (2) To reduce key space overheads, MLK uses file-level key to encrypt chunk-level keys so that the key space will not increase with the number of sharing users. Furthermore, MLK splits the file-level keys into share-level keys and distributes them to multiple key servers to ensure security and reliability of file-level keys. Our security analysis demonstrates that SecDep ensures data confidentiality and key security. Our experiment results based on several large real-world datasets show that SecDep is more time-efficient and key-space-efficient than the state-of-the-art secure deduplication approaches.

Data deduplication has gained increasing attention and popularity as a space-efficient approach in backup storage systems. One of the main challenges for centralized data deduplication is the scalability of fingerprint-index search. In this paper, we propose SiLo, a near-exact and scalable deduplication system that effectively and complementarily exploits similarity and locality of data streams to achieve high duplicate elimination, throughput, and well balanced load at extremely low RAM overhead. The main idea behind SiLo is to expose and exploit more similarity by grouping strongly correlated small files into a segment and segmenting large files, and to leverage the locality in the data stream by grouping contiguous segments into blocks to capture similar and duplicate data missed by the probabilistic similarity detection. SiLo also employs a locality based stateless routing algorithm to parallelize and distribute data blocks to multiple backup nodes. By judiciously enhancing similarity through the exploitation of locality and vice versa, SiLo is able to significantly reduce RAM usage for index-lookup, achieve the near-exact efficiency of duplicate elimination, maintain a high deduplication throughput, and obtain load balance among backup nodes.

Data reduction has become increasingly important in storage systems due to the explosive growth of digital data in the world that has ushered in the big data era. In this paper, we present DARE, a Deduplication-Aware Resemblance detection and Elimination scheme for compressing backup datasets that effectively combines data deduplication and delta compression to achieve high data reduction efficiency at low overhead. The main idea behind DARE is to employ a scheme, call Duplicate-Adjacency based Resemblance Detection (DupAdj), by considering any two data chunks to be similar (i.e., candidates for delta compression) if their respective adjacent data chunks are found to be duplicate in a deduplication system, and then further enhance the resemblance detection efficiency by an improved super-feature approach. Our experimental results based on real-world and synthetic backup datasets show that DARE achieves an additional data reduction by a factor of more than 2 (2X) on top of deduplication with very low overhead while nearly doubling the data restore performance of deduplication-only systems by supplementing delta compression to deduplication.

Data deduplication, an efficient space reduction method, has gained increasing attention and popularity in data-intensive storage systems. Most existing state-of-the-art deduplication methods remove redundant data at either the file level or the chunk level, which incurs unavoidable and significant overheads in time (due to chunking and fingerprinting). These overheads can degrade the write performance to an unacceptable level in a data storage system. In this paper, we propose P-Dedupe, a fast and scalable deduplication system. The main idea behind P-Dedupe is to fully compose pipelined and parallel computations of data deduplication by effectively exploiting the idle resources of modern computer systems with multi-core and many-core processor architectures. Our experimental evaluation of the P-Dedupe prototype based on real-world datasets shows that P-Dedupe speeds up the deduplication write throughput by a factor of 2~4 through pipelining deduplication and parallelizing hash calculation and achieves 80%~250% of the performance of a conventional storage system without data deduplication.