Todays high-end parallel clusters are architecturally very complex. Most large scale applications nowadays are utilizing multiple parallel programming paradigms to achieve the required scalability, with MPI+threads being the most common approach. Yet, as of today, there is no parallel I/O library that matches this hybrid programming model. File I/O operations are typically executed by a single thread for each process. This paper explores multi-threaded optimizations for individual MPI I/O operations, an important step towards matching the execution model of modern parallel applications. We describe the changes necessary to the internal processing in the MPI I/O library as well as to the file access phase. We demonstrate the performance improvement of the redesigned functions using multiple benchmarks and on multiple platforms for many scenarios over the original, single-threaded version.

Many scientific applications operate on data sets that span hundreds of Gigabytes or even Terabytes in size. Large data sets often use compression to reduce the size of the files. Yet as of today, parallel I/O libraries do not support reading and writing compressed files, necessitating either expensive sequential compression/decompression operations before/after the simulation, or omitting advanced features of parallel I/O libraries, such as collective I/O operations. This paper introduces parallel I/O on compressed data files, discusses the key challenges, requirements, and solutions for supporting compressed data files in MPI I/O, as well as limitations on some MPI I/O operations when using compressed data files. The paper details handling of individual read and write operations of compressed data files, and presents an extension to the two-phase collective I/O algorithm to support data compression. The paper further presents and evaluates an implementation based on the Snappy compression library and the OMPIO parallel I/O framework. The performance evaluation using multiple data sets demonstrate significant performance benefits when using data compression on a parallel BeeGFS file system.

Summary Volunteer PC hosts represent massive computation capacity at a low cost but are challenging to employ for general parallel computing. This paper presents the design, execution model, implementation, and evaluation of the Volpex framework for robust execution of parallel codes on volunteer PC grids characterized by system and network heterogeneity, varying availability, and frequent failures. The communication model is based on one‐sided Put/Get calls to an abstract global shared space enhanced to support multiple autonomous instances of the same process at different stages of execution. Our approach customizes and combines the use of replication, checkpointing, and host selection. This presents formidable challenges that are addressed in this work; efficient checkpointing of distributed replicated processes, dynamic management of redundancy, quick restart in a distributed environment, and application specific host selection. The integrated runtime system is shown to effectively execute moderate size, coarse‐grain, communicating codes on a worldwide distributed volunteer environment, a new milestone in volunteer computing. Extensive evaluation is conducted with example scientific codes on a pool of around 600 volunteer hosts. The results demonstrate the trade‐offs in deploying checkpointing, redundancy, and host selection, and how these methods combine to provide application performance that is close to the ideal failure free performance.

Many any large scale scientific applications spend a significant amount of time in file I/O operations. Collective I/O APIs provide higher level abstractions of I/O across a group of processes. They often reduce the time spent in file I/O by reorganizing data across processes to match the layout of the data on the file system. In this paper we present performance models for the communication occurring in collective write operations as a first step towards developing a full and accurate mode of collective I/O operations. The models derived in this paper take both the application data decomposition and the file domain partitioning strategies used by the I/O library into account. We discuss properties of our performance models and demonstrate using LogGP parameters derived on multiple platforms their impact on the performance of collective I/O operations. The paper further provides comparison to actual measurements performed on an InfiniBand cluster. Our results indicate a good overall match between predicted and observed behavior.

Due to the growing size of compute clusters, large scale parallel applications increasingly have to deal with hardware malfunctions and other failure scenarios during execution. The overall goal of this research is to get good performance of MapReduce applications despite failures. The paper focuses on evaluation of the performance of two representative Hadoop MapReduce applications, 'WordCount' and 'Stack Exchange', with different execution parameters and under different failure scenarios. The paper also presents different options to inject failures into MapReduce applications to simulate real world failures. Some of the preliminary observations are that slowdown due to failure is higher with relatively larger input split sizes, slowdown peaks near the optimal split size, and that the performance and slowdown are sensitive to key MapReduce execution parameters.

The time spent in communication operations is a major factor in determining the scalability of parallel applications. Tuning the parameters of a communication library can be used to adapt its characteristics to a particular platform, minimizing the communication time of an application. The goal of this paper is to improve theoretical and practical understanding of how performance improvements of point-to-point operations propagate to collective communication operations. We derive formulas to determine the expected improvement of a collective operation based on the improvement observed for a point-to-point communication using Hockney's model and the LogGP model. Our results indicate that many collective algorithms will inherently see a lower performance improvements compared to the improvement observed for point-to-point operations. Our evaluation shows for most test cases a good match between the predictions made by our models and the observed data, but also identifies multiple reasons for potential disparity between theory and practice.

Mapping of MPI processes to the available resources is an increasingly complex but important task on modern parallel systems. This paper presents a new approach to optimize the process placement of a parallel application based on its I/O access pattern. The paper introduces the SetMatch process mapping algorithm, which significantly reduces the cost of the communication occurring in collective I/O operations. The effectiveness of the approach has been evaluated for multiple scenarios on a PVFS2 file system. Our results demonstrate significant improvements in the communication time of collective I/O operations as well as improvements in the overall application execution time with our mapping strategy. The generalized SetMatch algorithm was the only mapping strategy that was able to provide adequate performance for all scenarios used in this paper.

A pool of distributed volunteer PCs presents an extremely hostile environment for execution of communicating parallel codes due to system and network heterogeneity, varying availability, and frequent failures. Well known methods for fault tolerance, specifically replication and check pointing, are challenging to deploy and not sufficient individually to provide continuous forward application progress. As the failure of a single logical process leads to application failure, the degree of redundancy needed for long running applications is too large to be practical. Check pointing and rollback does not provide protection against slow and variable speed nodes and is impractical when system wide MTBF is in minutes or less, common for a moderate size volunteer computing pool. The approach taken in this research is to exploit both, but that presents formidable challenges, efficient check pointing of distributed replicated processes, dynamic management of redundancy, quick restart in a distributed environment, and others. Proposed solution also leverages node selection based on availability prediction. The integrated runtime system is shown to effectively execute moderate size, coarse grain, communicating codes on a worldwide distributed volunteer environment, a new milestone in volunteer computing. The results provide new insight into how multiple techniques interact and contribute to robustness. The programming model is based on one-sided Put/Get calls to an abstract global shared space that works seamlessly with replicated processes. A Replica Exchange Molecular Dynamics code is employed to drive evaluation. The execution environment includes hosts on a University campus as well as hosts distributed around the world.

Volunteer PC grids represent massive computation capacity at a low cost, but are challenging to employ for parallel computing because of variable and unpredictable performance and availability. A communicating parallel program must employ explicit redundancy, or implicit redundancy with uncoordinated checkpoint-restart to make continuous forward progress in such an unreliable environment. A communication model based on one-sided Put/Get calls to an abstract global shared space is a good match as processes can execute their communication operations independently and asynchronously. However, no existing system is designed for redundant communicating processes. The key problem is that a single logical operation that impacts the global program state may be executed by different instances of the same process at different times leading to semantic inconsistency. This paper presents the design, execution model, implementation, and usage of {\em Volpex}, a communication layer for robust execution on volunteer PC grids. The research leads to a practical way to employ idle PCs for latency tolerant parallel computing applications.

Optimizing collective I/O operations is of paramount importance for many data intensive high performance computing applications. Despite the large number of algorithms published in the field, most current approaches focus on tuning every single application scenario separately and do not offer a consistent and automatic method of identifying internal parameters for collective I/O algorithms. Most notably, published work exists to optimize the number of processes actually touching a file, the so-called aggregators. This paper introduces a novel runtime approach to determine the number of aggregator processes to be used in a collective I/O operation depending on the file view, process topology, the per-process write saturation point, and the actual amount of data written in a collective write operation. The algorithm is evaluated on two different file systems with multiple benchmarks. In more than 80% of the test cases, our algorithm delivered a performance close to the best performance obtained by hand-tuning the number of aggregators for each scenario.

Idle desktops have been successfully used to run sequential and master-slave task parallel codes on a large scale in the context of volunteer computing. However, execution of message passing parallel programs in such environments is challenging because a pool of nodes to execute an application may have architectural and operating system heterogeneity, can include widely distributed nodes across security domains, and nodes may become unavailable for computation frequently and without warning. The VolPEx (Parallel Execution on Volatile Nodes) tool set is building MPI support in such environments based on selective use of process redundancy and message logging. However, addressing this challenge requires tradeoffs between performance, portability, and usability. The paper introduces a robust MPI library that is designed to be highly portable across heterogeneous architectures and operating systems. This VolpexPyMPI library is built with Python, works with Linux and Windows platforms and accepts user level MPI programs written in C or FORTRAN. The performance of VolpexPyMPI is compared with a traditional C based implementation of MPI. The paper examines in detail the tradeoffs of these usability focused and performance focused approaches.

Abstract The Abstract Data and Communication Library (ADCL) is an adaptive communication library optimizing application level group communication operations at runtime. The library provides for a given communication pattern a large number of implementations and incorporates a runtime selection logic in order to choose the implementation leading to the highest performance. In this paper, we demonstrate how an application utilizing ADCL is deployed on a wide range of HPC architectures, including an IBM Blue Gene/L, an NEC SX‐8, an IBM Power PC cluster using an IBM Federation Switch, an AMD Opteron cluster utilizing a 4xInfiniBand and a Gigabit Ethernet network, and an Intel EM64T cluster using a hierarchical Gigabit Ethernet network with reduced uplink bandwidth. We demonstrate, how different implementations for the three‐dimensional neighborhood communication lead to the minimal execution time of the application on different architectures. ADCL gives the user the advantage of having to maintain only a single version of the source code and still have the ability to achieve close to optimal performance for the application on all architectures. Copyright © 2010 John Wiley & Sons, Ltd.

When an adaptive software component is employed to select the best-performing implementation for a communication operation at runtime, the correctness of the decision taken strongly depends on detecting and removing outliers in the data used for the comparison. This automatic decision is greatly complicated by the fact that the types and quantities of outliers depend on the network interconnect and the nodes assigned to the job by the batch scheduler. This paper evaluates four different statistical methods used for handling outliers, namely a standard interquartile range method, a heuristic derived from the trimmed mean value, cluster analysis and a method using robust statistics. Using performance data from the Abstract Data and Communication Library (ADCL) we evaluate the correctness of the decisions made with each statistical approach over three fundamentally different network interconnects, namely a highly reliable InfiniBand network, a gigabit Ethernet network having a larger variance in the performance, and a hierarchical gigabit Ethernet network.

This paper introduces the Abstract Data and Communication Library (ADCL). ADCL is an application level communication library aiming at providing the highest possible performance for application level communication operations on a given execution environment. The library provides for each communication pattern a large number of implementations and incorporates a runtime selection logic in order to choose the implementation leading to the highest performance of the application on the current platform. Two different runtime selection algorithms are currently available within ADCL: the library can either apply a brute force search strategy which tests all available implementations of a given communication pattern; alternatively, a heuristic relying on attributes characterizing an implementation has been developed in order to speed up the runtime decision procedure. The paper also evaluates the performance of a finite difference code using ADCL on an AMD Opteron cluster using InfiniBand and Gigabit Ethernet interconnects.

As the number of processors in today's high performance computers continues to grow, the mean-time-to-failure of these computers are becoming significantly shorter than the execution time of many current high performance computing applications. Although today's architectures are usually robust enough to survive node failures without suffering complete system failure, most today's high performance computing applications can not survive node failures and, therefore, whenever a node fails, have to abort themselves and restart from the beginning or a stable-storage-based checkpoint.This paper explores the use of the floating-point arithmetic coding approach to build fault survivable high performance computing applications so that they can adapt to node failures without aborting themselves. Despite the use of erasure codes over Galois field has been theoretically attempted before in diskless checkpointing, few actual implementations exist. This probably derives from concerns related to both the efficiency and the complexity of implementing such codes in high performance computing applications. In this paper, we introduce the simple but efficient floating-point arithmetic coding approach into diskless checkpointing and address the associated round-off error issue. We also implement a floating-point arithmetic version of the Reed-Solomon coding scheme into a conjugate gradient equation solver and evaluate both the performance and the numerical impact of this scheme. Experimental results demonstrate that the proposed floating-point arithmetic coding approach is able to survive a small number of simultaneous node failures with low performance overhead and little numerical impact.

Distributing an application onto several machines is one of the key aspects of Grid-computing. In the last few years several groups have developed solutions for the occurring communication problems. However, the focus on the machines used in distributed environments has changed over the time from Massively Parallel Processing Systems to Clusters. This paper presents the problems arising when coupling several cluster like systems and discusses possible solutions. Furthermore, we present performance results and performance drawbacks of the solutions discussed before.

Abstract Grid computing has become a popular concept in the last few years. While in the beginning the driving force was metacomputing, the focus has now shifted towards resource management issues and concepts like ubiquitous computing. For the High‐Performance Computing Center Stuttgart (HLRS) the key challenges of Grid computing have come from the demands of its users and customers. With high‐speed networks in place, programmers expect to be able to exploit the overall performance of several instruments and high‐speed systems for their applications. In order to meet these demands, HLRS has set out a research effort to provide these users with the necessary tools to develop and run their codes on clusters of supercomputers. This has resulted in the development of a basic Grid‐computing environment for technical and scientific computing. In this paper we describe the building blocks of this software development environment and focus specifically on communication and debugging. We present the Grid‐enabled MPI implementation PACX‐MPI and the MPI debugger MARMOT. Copyright © 2002 John Wiley & Sons, Ltd.

A variety of projects worldwide are developing what we call "heterogeneous MPI". These MPI implementations are designed to operate on multiple computers, perhaps of different types, ranging in complexity from a set of desktop workstations to several supercomputers connected via a wide area network. These considerations led us to investigate the feasibility of defining a common API that could be used within MPI implementations to access process startup, initialization, monitoring, and control functions provided by an underlying process management system. If various MPI implementations coded to that API, one could then develop multiple "process management" modules that could be reused within different MPI implementations, thus allowing partitioning of effort between different development groups. In pursuit of this goal, we have designed such an API, which we call BNR. The major goals of the BNR interface are outlined.

Coupling computer resources to work on a given problem has been a successful strategy for years. Due to the existence of high bandwidth WAN it has become possible to couple big machines to build clusters that outperform the most powerful existing single computers. The article discusses such a powerful cluster, a metacomputer. While for MPPs the message passing model was standardized already 4 years ago (1995), a standard for interoperable MPI has only been published recently. We briefly present an approach for MPI based metacomputing called PACX-MPI. We also describe other approaches, and give an overview of the technical concept.