A detailed understanding of HPC application's resource needs and their complex interactions with each other and HPC platform resources is critical to achieving scalability and performance. Such understanding has been difficult to achieve because typical application profiling tools do not capture the behaviors of codes under the potentially wide spectrum of actual production conditions and because typical monitoring tools do not capture system resource usage information with high enough fidelity to gain sufficient insight into application performance and demands. In this paper we present both system and application profiling results based on data obtained through synchronized system wide monitoring on a production HPC cluster at Sandia National Laboratories (SNL). We demonstrate analytic and visualization techniques that we are using to characterize application and system resource usage under production conditions for better understanding of application resource needs. Our goals are to improve application performance (through understanding application-to-resource mapping and system throughput) and to ensure that future system capabilities match their intended workloads.

Understanding how resources of High Performance Compute platforms are utilized by applications both individually and as a composite is key to application and platform performance. Typical system monitoring tools do not provide sufficient fidelity while application profiling tools do not capture the complex interplay between applications competing for shared resources. To gain new insights, monitoring tools must run continuously, system wide, at frequencies appropriate to the metrics of interest while having minimal impact on application performance. We introduce the Lightweight Distributed Metric Service for scalable, lightweight monitoring of large scale computing systems and applications. We describe issues and constraints guiding deployment in Sandia National Laboratories' capacity computing environment and on the National Center for Supercomputing Applications' Blue Waters platform including motivations, metrics of choice, and requirements relating to the scale and specialized nature of Blue Waters. We address monitoring overhead and impact on application performance and provide illustrative profiling results.

Abstract In 2007, the Cray Red Storm system at Sandia National Laboratories completed an upgrade of the processor and network hardware. Single‐core 2.0 GHz AMD Opteron processors were replaced with dual‐core 2.4 GHz AMD Opterons, while the network interface hardware was upgraded from a sustained rate of 1.1 GBps to 2.0 GBps (without changing the router link rates). These changes more than doubled the theoretical peak floating‐point performance of the compute nodes and doubled the bandwidth performance of the network. This paper provides an analysis of the impact of this upgrade on the performance of several applications and micro‐benchmarks. Performance results show that the additional core provides a performance boost of 20–50% for real applications on a fixed problem size per‐socket basis on up to 2048 cores and that scalability is impacted relatively little by the upgrade. Copyright © 2009 John Wiley & Sons, Ltd.