Compression ratio estimation is an important optimization of I/O workflows processing terabytes of data. Applications such as compression auto-tuning or lossy compressor selection require a high-throughput, accurate estimation to be fast. Prior works that utilize sampling are fast but inaccurate, while approaches that do not use sampling are accurate but slow. We present a novel, lightweight sampling technique that excels in speed and accuracy, by leveraging both statistical and spatial properties of dataset samples. Through a comprehensive sensitivity analysis we show that there is no one-size-fits-all sampling strategy because of differences in compression principles, and accounting for these differences has a direct impact on overall prediction accuracy in the methods that use these predictions. Our estimation technique demonstrates superior ($1.36 \times$to$11 \times$) prediction accuracy compared to existing sampling methods while significantly improving data throughput by$3.46 \times$compared to non-sampling approaches.

Abstract Growing constraints on memory utilization, power consumption, and I/O throughput have increasingly become limiting factors to the advancement of high performance computing (HPC) and edge computing applications. IEEE‐754 floating‐point types have been the de facto standard for floating‐point number systems for decades, but the drawbacks of this numerical representation leave much to be desired. Alternative representations are gaining traction, both in HPC and machine learning environments. Posits have recently been proposed as a drop‐in replacement for the IEEE‐754 floating‐point representation. We survey the state‐of‐the‐art and state‐of‐the‐practice in the development and use of posits in edge computing and HPC. The current literature supports posits as a promising alternative to traditional floating‐point systems, both as a stand‐alone replacement and in a mixed‐precision environment. Development and standardization of the posit type is ongoing, and much research remains to explore the application of posits in different domains, how to best implement them in hardware, and where they fit with other numerical representations.

Progress in high-performance computing (HPC) systems has led to complex applications that stress the I/O subsystem by creating vast amounts of data. Lossy compression reduces data size considerably, but a single error renders lossy compressed data unusable. This sensitivity stems from the high information content per bit in compressed data and is a critical issue as soft errors that cause bit-flips have become increasingly commonplace in HPC systems. While many works have improved lossy compressor performance, few have sought to address this critical weakness.