We examine the I/O behavior of thousands of supercomputing applications "in the wild," by analyzing the Darshan logs of over a million jobs representing a combined total of six years of I/O behavior across three leading high-performance computing platforms. We mined these logs to analyze the I/O behavior of applications across all their runs on a platform; the evolution of an application's I/O behavior across time, and across platforms; and the I/O behavior of a platform's entire workload. Our analysis techniques can help developers and platform owners improve I/O performance and I/O system utilization, by quickly identifying underperforming applications and offering early intervention to save system resources. We summarize our observations regarding how jobs perform I/O and the throughput they attain in practice.

I/O has become one of the determining factors of HPC application performance. Understanding an application's I/O activity requires a multi-level view of the I/O function flow that includes high-level I/O libraries. We have developed a tracing framework, called Recorder, that captures I/O function calls at multiple layers of the parallel I/O stack without requiring source code modifications. In this paper, we show how Recorder's trace output can be used to investigate I/O activity and identify performance inefficiencies in two I/O benchmarks running on a leading edge HPC platform. Future work to organize and present the collected information more intuitively will further increase the value of Recorder's capabilities. We believe that a multi-level I/O tracing framework can provide key insights to end users and I/O library developers working to improve I/O on HPC platforms.

We present an auto-tuning system for optimizing I/O performance of HDF5 applications and demonstrate its value across platforms, applications, and at scale. The system uses a genetic algorithm to search a large space of tunable parameters and to identify effective settings at all layers of the parallel I/O stack. The parameter settings are applied transparently by the auto-tuning system via dynamically intercepted HDF5 calls.