High performance computing resources attract a wide range of computational users and corresponding job widths and lengths. For example, on the petaflop Cray XT5 machine, Kraken, users submit jobs ranging from a few hundred cores (capacity computing) to over hundred thousand cores (capability computing). Traditionally it has been difficult to maintain high utilization while juggling such a diverse job mix. This paper explores four unique approaches to achieve our scheduling goals of maximizing utilization on four distinct resources at the National Institute for Computational Sciences. The resources include the petaflop machine, Kraken, Athena - a 166 TF Cray XT4, a 4 TB shared memory NUMA machine called Nautilus, and a GPU cluster called Keeneland.

We investigate here, both theoretically and by demonstration, scaling file storage to the very widest possible extents. We use IBM's GPFS file system, with extensions developed by the San Diego Supercomputer Center in collaboration with IBM. Geographically, the file system extends across the United States, including Pittsburgh, Illinois, and San Diego, California, with the TeraGrid 40 Gb/s backbone providing the wide area network connectivity. We show the results from two demonstrations, at each of the past two supercomputing conferences, SC03 in Phoenix, Arizona, and SC04 in Pittsburgh, Pennsylvania. The second demonstration was purposely designed to presage an intended production facility across the National Science Foundation's TeraGrid.

In this paper we describe the evolution of Global File Systems from the concept of a few years ago, to a first demonstration using hardware Fibre Channel frame encoding into IP packets, to a native GFS, to a full prototype demonstration, and finally to a production implementation. The surprisingly excellent performance of the Global File Systems over standard TCP/IP Wide Area Networks has made them a viable candidate for the support of Grid Supercomputing. The implementation designs and performance results are documented within this paper. We also motivate and describe the authentication extensions we made to the IBM GPFS file system, in collaboration with IBM. In several ways Global File Systems are superior to the original approach of wholesale file movement between grid sites and we speculate as to future modes of operation.