Over the past 7 years, the San Diego Supercomputer Center has worked to produce the highest possible performance file systems available to the National Science Foundation community in the USA. Most of this was done with GPFS, IBM's parallel file system, but several distinctly different configurations were designed and implemented with numerous lessons learned in the process. All of these systems provided transfer rates in the multiple GB/s range. In this paper, we detail the configurations and their intended modes of operation and, as much as possible, show the resulting performance. We attempt to describe the advantages and disadvantages of each approach with an emphasis on the implications for future systems

High Performance Computing E-Science has numerous requirements well in excess of the normal environment sufficient for routine computations. The data requirements in particular may be in the multiterabyte regime, with transfer rates in the several Gb/s or more range. Such data capabilities may only be available at a single location. On the other hand, understanding the data, via data-mining and visualization may require completely different facilities, while the actual data production may be only possible at yet a third location. The use of Global File System with multi-Gb/s speeds and hundreds of TeraBytes capacity can help the scientific researchers, but the simultaneous utilization of several systems also requires a reasonably sophisticated co-scheduling capability. In this paper, we show how the TeraGrid is combining massive computational clusters, a Global File System, very powerful visualization systems, and a co-scheduler to enable massive E-Science in a very coordinated and usable manner.

Over the past decade, computing power has increased extraordinarily, enabling researchers to tackle large complex problems -- a great step forward in our ability to model the complex world. This increased capability resulted in massive amounts of data being produced and used by increasingly detailed simulations. It is the location, viewing, manipulation, storage, movement, sharing, and interpretation of this data -- now in hundreds of terabytes and growing -- that is causing major performance bottleneck. Data Intensive Computing has emerged as a topic to address the problem. The Advanced Virtual Engine Test Cell (AVETeC) Inc. and Computer Sciences Corporation (CSC) are under contract to establish, integrate and manage a test bed located at Department of Defense Major Shared Resource Center, Wright-Patterson Air Force Base, OH, at AVETeC in Springfield, OH collaborating with Department of Energy, and NASA Goddard, Greenbelt, MD. This test bed environment will actively evaluate emerging technologies to improve data management.

We investigate here, both theoretically and by demonstration, scaling file storage to the very widest possible extents. We use IBM's GPFS file system, with extensions developed by the San Diego Supercomputer Center in collaboration with IBM. Geographically, the file system extends across the United States, including Pittsburgh, Illinois, and San Diego, California, with the TeraGrid 40 Gb/s backbone providing the wide area network connectivity. We show the results from two demonstrations, at each of the past two supercomputing conferences, SC03 in Phoenix, Arizona, and SC04 in Pittsburgh, Pennsylvania. The second demonstration was purposely designed to presage an intended production facility across the National Science Foundation's TeraGrid.

In this paper we describe the evolution of Global File Systems from the concept of a few years ago, to a first demonstration using hardware Fibre Channel frame encoding into IP packets, to a native GFS, to a full prototype demonstration, and finally to a production implementation. The surprisingly excellent performance of the Global File Systems over standard TCP/IP Wide Area Networks has made them a viable candidate for the support of Grid Supercomputing. The implementation designs and performance results are documented within this paper. We also motivate and describe the authentication extensions we made to the IBM GPFS file system, in collaboration with IBM. In several ways Global File Systems are superior to the original approach of wholesale file movement between grid sites and we speculate as to future modes of operation.

The primary design goal for P3D has been to produce a metafile format which can support the needs of scientific visualization while remaining compatible with a wide range of common renderers. To this end, rendering features which are available on only a few renderers and cannot easily be emulated have been excluded from P3D. P3D is based on the Common Lisp programming language with a small number of extensions for the description of geometry. The programmability of P3D greatly eases the task of adapting existing model generators to produce P3D, in the same fashion that the programmability of the Postscript page description language eases the task of modifying code to generate Postscript. Further, programmability allows knowledge of the particular scientific domain of the model to be incorporated into the metafile. This method provides easy representation of directed acyclic graphs and complete programmability to facilitate interfaces to new model generators, without the introduction of a new and unfamiliar language.>