Summary In order to provide a stepping stone from the Argonne Leadership Computing Facility's (ALCF) world class production 10 petaFLOP IBM BlueGene/Q system, Mira, to its next generation 200 petaFLOPS 3rd generation Intel Xeon Phi system, Aurora, ALCF worked with Intel and Cray to acquire an 8.6 petaFLOPS 2nd generation Intel Xeon Phi–based system named Theta. Theta was delivered, installed, integrated, and accepted on an aggressive schedule in just over 3 months. We will detail how we were able to successfully meet the aggressive deadline as well as lessons learned during the process.

We present an overview of the current status of input/output (I/O) on the Cray XT line of supercomputers and provide guidance to application developers and users for achieving efficient I/O. Many I/O benchmark results are presented, based on projected I/O requirements for some widely used scientific applications in the Department of Energy. Finally, we interpret and summarize these benchmark results to provide forward-looking guidance for I/O in large-scale application runs on a Cray XT3/XT4.

BlueGene/P (BG/P) is the second generation BlueGene architecture from IBM, succeeding BlueGene/L (BG/L). BG/P is a system-on-a-chip (SoC) design that uses four PowerPC 450 cores operating at 850 MHz with a double precision, dual pipe floating point unit per core. These chips are connected with multiple interconnection networks including a 3-D torus, a global collective network, and a global barrier network. The design is intended to provide a highly scalable, physically dense system with relatively low power requirements per flop. In this paper, we report on our examination of BG/P, presented in the context of a set of important scientific applications, and as compared to other major large scale supercomputers in use today. Our investigation confirms that BG/P has good scalability with an expected lower performance per processor when compared to the Cray XT4's Opteron. We also find that BG/P uses very low power per floating point operation for certain kernels, yet it has less of a power advantage when considering science driven metrics for mission applications.

The scientific simulation capabilities of next generation high-end computing technology will depend on striking a balance among memory, processor, I/O, and local and global network performance across the breadth of the scientific simulation space. The Cray XT4 combines commodity AMD dual core Opteron processor technology with the second generation of Cray's custom communication accelerator in a system design whose balance is claimed to be driven by the demands of scientific simulation. This paper presents an evaluation of the Cray XT4 using micro-benchmarks to develop a controlled understanding of individual system components, providing the context for analyzing and comprehending the performance of several petascale-ready applications. Results gathered from several strategic application domains are compared with observations on the previous generation Cray XT3 and other high-end computing systems, demonstrating performance improvements across a wide variety of application benchmark problems.

Oak Ridge National Laboratory recently received delivery of a 5,294 processor Cray XT3. The XT3 is Cray's third-generation massively parallel processing system. The system builds on a single processor node - built around the AMD Opteron - and uses a custom chip - called SeaStar - to provide interprocess or communication. In addition, the system uses a lightweight operating system on the compute nodes. This paper describes our initial experiences with the system, including micro-benchmark, kernel, and application benchmark results. In particular, we provide performance results for strategic Department of Energy applications areas including climate and fusion. We demonstrate experiments on the installed system, scaling applications up to 4,096 processors.

We will show the performance history of the fusion code GYRO on two diverse architectures, namely a Cray XT3 and a Cray X1E. The history shows performance of selected versions of GYRO for two standard benchmarks. For this comparison, we choose one compiler version on each machine and keep it constant for all the tests and run the tests over a short time frame for consistency. We will present Cray X1E and Cray XT3 results; we expect to present dual-core XT3 results as well, time-permitting, following an upgrade this summer. The contribution of this work will be a summary of the performance enhancements that are portable to three classes of machines: vector, scalar, and multi-core scalar. Getting good performance on modern HPC systems is more than just finding the right compiler options. Knowing what performance optimizations are portable across a variety of machines is essential.

GYRO solves the 5-dimensional gyrokinetic-Maxwell equations in shaped plasma geometry, using either a local (fluxtube) or global radial domain. It has been ported to a variety of modern MPP platforms including a number of commodity clusters, IBM SPs and the Cray X1. We have been able to quickly design and analyze new physics scenarios in record time using the Cray X1: (i) transport barrier studies (Phys. Plasmas 11 (2004) 1879), (ii) the local limit of global simulations (Phys. Plasmas 11 (2004) L25), (iii) kinetic electron and finite-beta generalizations of a community-wide benchmark case, and (iv) impurity transport with application to fuel separation in burning D-T plasmas (to be submitted to Nuclear Fusion). We report on recent physics progress and studies. Further, we discuss GYRO performance across several architectures.

Oak Ridge National Laboratory installed a 32 processor Cray X1 in March, 2003, and will have a 256 processor system installed by October, 2003. In this paper we describe our initial evaluation of the X1 architecture, focusing on microbenchmarks, kernels, and application codes that highlight the performance characteristics of the X1 architecture and indicate how to use the system most efficiently.

A code for computing the eigenvalues of a complex Hessenberg matrix is presented. This code computes the Schur decomposition of a complex Hessenberg matrix. Together with existing ScaLAPACK routines, the eigenvalues of dense complex matrices can be directly computed using a parallel QR algorithm.This parallel complex Schur decomposition routine was developed to fill a void in the ScaLAPACK library and was based on the parallel real Schur decomposition routine already in ScaLAPACK. The real-arithmetic version was appropriately modified to make it work with complex arithmetic and implement a complex multiple bulge QR algorithm. This also required the development of new auxiliary routines that perform essential operations for the complex Schur decomposition, and that will provide additional linear algebra computation capability to the parallel numerical library community.

Oak Ridge National Laboratory recently received 27 32-way IBM pSeries 690 SMP nodes. In this paper, we describe our initial evaluation of the p690 architecture, focusing on the performance of benchmarks and applications that are representative of the expected production workload.