Summary form only given. A growing trend in developing large and complex applications on today's Teraflop computers is to integrate stand-alone and/or semiindependent program components into a comprehensive simulation package. One example is the climate system model, which consists of atmosphere, ocean, land-surface and sea-ice. Each component is semiindependent and has been developed at different institutions. We study how this multicomponent multiexecutable application can run effectively on distributed memory architectures. We identify five effective execution modes and develop the MPH library to support application developments utilizing these modes. MPH performs component-name registration, resource allocation and initial component handshaking in a flexible way.

We investigate remapping multi-dimensional arrays on cluster of SMP architectures under OpenMP, MPI, and hybrid paradigms. Traditional method of array transpose needs an auxiliary array of the same size and a copy back stage. We recently developed an in-place method using vacancy tracking cycles. The vacancy tracking algorithm outperforms the traditional 2-array method as demonstrated by extensive comparisons. The independence of vacancy tracking cycles allows efficient parallelization of the in-place method on SMP architectures at node level. Performance of multi-threaded parallelism using OpenMP are tested with different scheduling methods and different number of threads. The vacancy tracking method is parallelized using several parallel paradigms. At node level, pure OpenMP outperforms pure MPI by a factor of 2.76. Across entire cluster of SMP nodes, the hybrid MPI/OpenMP implementation outperforms pure MPI by a factor of 4.44, demonstrating the validity of the parallel paradigm of mixing MPI with OpenMP.

In solving Partial Differential Equations, such as the Barotropic equations in ocean models, on Distributed Memory Computers, finite difference methods are commonly used. Most often, processor subdomain boundaries must be updated at each time step. This boundary update process involves many messages of small sizes, therefore large communication overhead. Here we propose a new approach which expands the ghost cell layers and thus updates boundaries much less frequently --- reducing total message volume and groupping small messages into bigger ones. Together with a technique for eliminating diagonal communications, the method speedup communication substantially, upto 170%. We explain the method and implementation in details, provide systematic timing results and performance analysis on the Cray T3E and IBM SP.

Numerical reproducibility and stability of large scale scientific simulations, especially climate modeling, on distributed memory parallel computers are becoming critical issues. In particular, global summation of distributed arrays is most susceptible to rounding errors, and their propagation and accumulation cause uncertainty in final simulation results. We analyzed several accurate summation methods and found that two methods are particularly effective to improve (ensure) reproducibility and stability: Kahan's self-compensated summation and Bailey's double-double precision summation. We provide an MPI operator MPLSUMDD to work with MPI collective operations to ensure a scalable implementation on large number of processors. The final methods are particularly simple to adopt in practical codes.

We describe an efficient and scalable parallel I/O strategy for writing out gigabytes of data generated hourly in the ocean model simulations on massively parallel distributed-memory architectures.Working with Modular Ocean Model, using netCDF file system, and implemented on Cray T3E, the strategy speeds up I/O by a factor of 50 in the sequential case.In parallel case, on 32 processors up to 512 processors, our implementation writes out most model dynamic fields of 969 MB to a single netCDF file in 65 seconds, independent of the number of processors.The remap-and-write parallel strategy resolves the memory limitation problem and requires minimal collective I/O capability of the file system.Several critical optimizations on memory management and file access are carried out, ensuring scalability and speeding up numerical simulation due to the improved memory organizations.