High-resolution climate simulations require tremendous computing resources and can generate massive datasets. At present, preserving the data from these simulations consumes vast storage resources at institutions such as the National Center for Atmospheric Research (NCAR). The historical data generation trends are economically unsustainable, and storage resources are already beginning to limit science objectives. To mitigate this problem, we investigate the use of data compression techniques on climate simulation data from the Community Earth System Model. Ultimately, to convince climate scientists to compress their simulation data, we must be able to demonstrate that the reconstructed data reveals the same mean climate as the original data, and this paper is a first step toward that goal. To that end, we develop an approach for verifying the climate data and use it to evaluate several compression algorithms. We find that the diversity of the climate data requires the individual treatment of variables, and, in doing so, the reconstructed data can fall within the natural variability of the system, while achieving compression rates of up to 5:1.

Summary form only given. Numerical computations have accelerated significantly since 2005 thanks to two complementary, silicon-enabled trends: multi-core processing and single instruction, multiple data (SIMD) accelerators. Unfortunately, due to fundamental limitations of physics, these two trends could not be accompanied by a corresponding increase in memory, storage, and I/O bandwidth. High-performance computing (HPC) is the proverbial “canary in the coal mine” of multi-core processing. When HPC hits a multi-core will likely encounter a similar limit in few years. We describe the computationally efficient (Fig 1b) and adaptive APplication AXceleration (APAX) numerical encoding method to reduce the memory wall for integers and floating-point operands. APAX achieves encoding rates between 3:1 and 10:1 without changing the dataset's statistical or spectral characteristics. APAX encoding takes advantage of three characteristics of all numerical sequences: peak-to-average ratio, oversampling, and effective number of bits (ENOB). Uncertainty quantification and spectral methods quantify the degree of uncertainty (accuracy) in numerical datasets. APAX profiler creates a rate-correlation graph with recommended operating signals, and fundamental limit, consumer point, provides 18 quantitative metrics comparing the original and decoded displays input and residual spectra with a residual histogram. On 24 integer and floating-point HPC datasets taken from climate, multi-physics, and seismic simulations, APAX averaged 7.95:1 encoding ratio at a Pearson's correlation coefficient of 0. 999948, and a spectral margin (input spectrum min - residual spectrum mean) of 24 dB. HPC scientists confirmed that APAX did not change HPC simulation results DRAM and disk transfers by 8x, accelerating HPC “time to results” by 20% while reducing to 50%.

This paper examines whether lossy compression can be used effectively in physics simulations as a possible strategy to combat the expected data-movement bottleneck in future high performance computing architectures. We show that, for the codes and simulations we tested, compression levels of 3--5X can be applied without causing significant changes to important physical quantities.

This paper describes the Samplify family of lossless and lossy real-time compression algorithms for FPGAs and ASICs, operating at rates up to 40 Gsamp/sec, which achieve efficient, cost-effective compression of between 2:1 and 8:1. Compression for measurements reduces bandwidth and storage requirements in automated test equipment (ATE), medical imaging, oscilloscope, and other mixed-signal test applications.