Commodity clusters augmented with application accelerators are evolving as competitive high performance computing systems. The Graphical Processing Unit (GPU) with a very high arithmetic density and performance per price ratio is a good platform for the scientific application acceleration. In addition to the interconnect bottlenecks among the cluster compute nodes, the cost of memory copies between the host and the GPU device have to be carefully amortized to improve the overall efficiency of the application. Scientific applications also rely on efficient implementation of the Basic Linear Algebra Subroutines (BLAS), among which the General Matrix Multiply (GEMM) is considered as the workhorse subroutine. In this paper, we study the performance of the memory copies and GEMM subroutines that are crucial to port the computational chemistry algorithms to the GPU clusters. To that end, a benchmark based on the NetPIPE [1] framework is developed to evaluate the latency and bandwidth of the memory copies between the host and the GPU device. The performance of the single and double precision GEMM subroutines from the NVIDIA CUBLAS 2.0 library are studied. The results have been compared with that of the BLAS routines from the Intel Math Kernel Library (MKL) to understand the computational trade-offs. The test bed is a Intel Xeon cluster equipped with NVIDIA Tesla GPUs.

General discussion about InfiniBand, what it is, how it works, and how the OpenFabrics driver stack provides the same software interface for both InfiniBand and 10gig RDMA ethernet nics. We will also include late breaking news about what we learned deploying a show-floor wide InfiniBand network as part of SCInet.

We present a novel approach for using high performance network attached parallel storage for out-of-core computation. Our approach utilizes many parallel disks and storage controllers with a near 1:1 ratio of compute nodes to storage servers. This, when combined with 30 Gigabit 12X InfiniBand interconnects, allows remote storage subsystem bandwidth to reach the same performance level as local memory-cache file I/O performance. This combination allows computational chemistry application problem sizes which require more than 100GB of intermediate data to store this data on disk without the disk I/O subsystem becoming the limiting factor. With sequential storage access speeds on the same order of magnitude as main memory performance, this allows out-of-core computation to become practical due to disk storage size being several orders of magnitude cheaper per GB than main memory storage.