Data visualization is typically a critical component of post-processing analysis workflows for floating-point output data from large simulation codes, such as global climate models. For example, images are often created from the raw data as a means for evaluation against a reference dataset or image. While the popular Structural Similarity Index Measure (SSIM) is a useful tool for such image comparisons, generating large numbers of images can be costly when simulation data volumes are substantial. In fact, computational cost considerations motivated our development of an alternative to the SSIM, which we refer to as the Data SSIM (DSSIM). The DSSIM is conceptually similar to the SSIM, but can be applied directly to the floating-point data as a means of assessing data quality. We present the DSSIM in the context of quantifying differences due to lossy compression on large volumes of simulation data from a popular climate model. Bypassing image creation results in a sizeable performance gain for this case study. In addition, we show that the DSSIM is useful in terms of avoiding plot-specific (but data-independent) choices that can affect the SSIM. While our work is motivated by and evaluated with climate model output data, the DSSIM may prove useful for other applications involving large volumes of simulation data.

Output data from modern Earth system model simulations are consuming increasingly massive amounts of storage resources, and storing these climate model data is not economically sustainable. Previous works have motivated lossy compression as a potential solution, which achieves greater compression ratios than lossless compression. This further reduction comes at the cost of a loss of information, and therefore, care must be taken to avoid introducing artifacts in the data that could affect scientific conclusions. In this paper we introduce a Python package designed to aid in the analysis of differences in large spatio-temporal datasets, such as those produced by global climate models. While the new package is agnostic to the source of the differences, our motivation is to enable climate scientists to more easily assess the effects of lossy data compression by visualizing and computing derived spatial-temporal quantities that compare lossily compressed datasets to the original dataset. Because Python is quickly becoming the tool of choice for scientific data analysis in the geoscience community, this new package makes use of the Python software stack in Pangeo (an active NSF-funded community platform for Big Data geoscience). Interoperability with other Pangeo software tools means that the new package easily integrates into climate scientists' post-processing and analysis workflows, which we hope will facilitate the adoption of lossy compression into the climate modeling community.

Large-scale simulation codes that model complicated science and engineering applications typically have huge and complex code bases. For such simulation codes, where bit-for-bit comparisons are too restrictive, finding the source of statistically significant discrepancies (e.g., from a previous version, alternative hardware or supporting software stack) in output is non-trivial at best. Although there are many tools for program comprehension through debugging or slicing, few (if any) scale to a model as large as the Community Earth System Model (CESM#8482;), which consists of more than 1.5 million lines of Fortran code. Currently for the CESM, we can easily determine whether a discrepancy exists in the output using a by now well-established statistical consistency testing tool. However, this tool provides no information as to the possible cause of the detected discrepancy, leaving developers in a seemingly impossible (and frustrating) situation. Therefore, our aim in this work is to provide the tools to enable developers to trace a problem detected through the CESM output to its source. To this end, our strategy is to reduce the search space for the root cause(s) to a tractable size via a series of techniques that include creating a directed graph of internal CESM variables, extracting a subgraph (using a form of hybrid program slicing), partitioning into communities, and ranking nodes by centrality. Runtime variable sampling then becomes feasible in this reduced search space. We demonstrate the utility of this process on multiple examples of CESM simulation output by illustrating how sampling can be performed as part of an efficient parallel iterative refinement procedure to locate error sources, including sensitivity to CPU instructions. By providing CESM developers with tools to identify and understand the reason for statistically distinct output, we have positively impacted the CESM software development cycle and, in particular, its focus on quality assurance.

Abstract Applying lossy data compression to climate model output is an attractive means of reducing the enormous volumes of data generated by climate models. However, because lossy data compression does not exactly preserve the original data, its application to scientific data must be done judiciously. To this end, a collection of measures is being developed to evaluate various aspects of lossy compression quality on climate model output. Given the importance of data visualization to climate scientists interacting with model output, any suite of measures must include a means of assessing whether images generated from the compressed model data are noticeably different from images based on the original model data. Therefore, in this work we conduct a forced‐choice visual evaluation study with climate model data that surveyed more than one hundred participants with domain relevant expertise. In addition to the images created from unaltered climate model data, study images are generated from model data that is subjected to two different types of lossy compression approaches and multiple levels (amounts) of compression. Study participants indicate whether a visual difference can be seen, with respect to the reference image, due to lossy compression effects. We assess the relationship between the perceptual scores from the user study to a number of common (full reference) image quality assessment (IQA) measures, and use statistical models to suggest appropriate measures and thresholds for evaluating lossily compressed climate data. We find the structural similarity index (SSIM) to perform the best, and our findings indicate that the threshold required for climate model data is much higher than previous findings in the literature.