In the last few decades, research techniques have improved lossless compression ratios by significantly increasing processing time. However, these techniques have not gained popularity in industry because production systems require high throughput and low resource utilization. Instead, real world improvements in compression are increasingly realized by building application-specific compressors which can exploit knowledge about the structure and semantics of the data being compressed. Application-specific compressor systems outperform even the best generic compressors, but these techniques have severe drawbacks -- they are inherently limited in applicability, are hard to develop, and are difficult to maintain and deploy. In this work, we show that these challenges can be overcome with a new compression strategy. We propose the "graph model" of compression, a new theoretical framework for representing compression as a directed acyclic graph of modular codecs. OpenZL implements this framework and compresses data into a self-describing wire format, any configuration of which can be decompressed by a universal decoder. OpenZL's design enables rapid development of application-specific compressors with minimal code. Experimental results demonstrate that OpenZL achieves superior compression ratios and speeds compared to state-of-the-art general-purpose compressors on a variety of real-world datasets. Compared to ratio-focused deep-learning compressors, OpenZL is competitive on ratio while being many orders of magnitude faster. Internal deployments at Meta have also shown consistent improvements in size and/or speed, with development timelines reduced from months to days. OpenZL thus represents a significant advance in practical, scalable, and maintainable data compression for modern data-intensive applications.

Data compression has emerged as a promising technique to alleviate the memory, storage, and network cost with some associated compute overheads in warehouse-scale datacenter services. Despite being one of the most important components of the overall datacenter taxes, there has not been a comprehensive characterization of compression usage in datacenter workloads. Such characterization is paramount for both compression software developers and hardware accelerator designers as it can help them make optimal design trade-offs decisions in terms of performance, power, and cost while meeting service-level agreements of target applications. Moreover, it can provide data-driven insights to application developers to find optimal compression configuration choices for their services. In this paper, we first provide a holistic characterization of compression as used by various warehouse-scale datacenter services at a global social media provider, Meta. Next, we deep dive into a few representative use cases of compression in the production environment and characterize compression usage of the services while running live traffic. Finally, we conduct sensitivity studies to understand how different compression configurations are relevant to the overall infrastructure cost, followed by future research directions for compression hardware and software development.

Data compression has emerged as a promising technique to alleviate the memory, storage, and network cost with some associated compute overheads in warehouse-scale datacenter services. Despite being one of the most important components of the overall datacenter taxes, there has not been a comprehensive characterization of compression usage in data center workloads. In this work, we first provide a holistic characterization of compression as used by various warehouse-scale datacenter services at a global social media provider (Meta). Next, we deep dive into a few representative use cases of compression in the production environment and characterize compression usage of services while running live traffic.