Relative debugging traces software errors by comparing two executions of a program concurrently - one code being a reference version and the other faulty. Relative debugging is particularly effective when code is migrated from one platform to another, and this is of significant interest for hybrid computer architectures containing CPUs accelerators or coprocessors. In this paper we extend relative debugging to support porting stencil computation on a hybrid computer. We describe a generic data model that allows programmers to examine the global state across different types of applications, including MPI/OpenMP, MPI/OpenACC, and UPC programs. We present case studies using a hybrid version of the `stellarator' particle simulation DELTA5D, on Titan at ORNL, and the UPC version of Shallow Water Equations on Crystal, an internal supercomputer of Cray. These case studies used up to 5,120 GPUs and 32,768 CPU cores to illustrate that the debugger is effective and practical.

Summary Contemporary parallel debuggers allow users to control more than one processing thread while supporting the same examination and visualisation operations of that of sequential debuggers. This approach restricts the use of parallel debuggers when it comes to large scale scientific applications run across hundreds of thousands compute cores. First, manually observing the runtime data to detect error becomes impractical because the data is too big. Second, performing expensive but useful debugging operations becomes infeasible as the computational codes become more complex, involving larger data structures, and as the machines become larger. This study explores the idea of a data‐centric debugging approach, which could be used to make parallel debuggers more powerful. It discusses the use ofad hocdebug‐time assertions that allow a user to reason about the state of a parallel computation. These assertions support the verification and validation of program state at runtime as a whole rather than focusing on that of only a single process state. Furthermore, the debugger's performance can be improved by exploiting the underlying parallel platform because the available compute cores can execute parallel debugging functions, while a program is idling at a breakpoint. We demonstrate the system with several case studies and evaluate the performance of the tool on a 20 000 cores Cray XE6. Copyright © 2013 John Wiley & Sons, Ltd.

Parallel debugging faces challenges in both scalability and efficiency. A number of advanced methods have been invented to improve the efficiency of parallel debugging. As the scale of system increases, these methods highly rely on a scalable communication protocol in order to be utilized in large-scale distributed environments. This paper describes a debugging middleware that provides fundamental debugging functions supporting multiple communication protocols. Its pluggable architecture allows users to select proper communication protocols as plug-ins for debugging on different platforms. It aims to be utilized by various advanced debugging technologies across different computing platforms. The performance of this debugging middleware is examined on a Cray XE Supercomputer with 21,760 CPU cores.

Traditional debuggers are of limited value for modern scientific codes that manipulate large complex data structures. This paper discusses a novel debug-time assertion, called a "Statistical Assertion", that allows a user to reason about large data structures, and the primitives are parallelised to provide an efficient solution. We present the design and implementation of statistical assertions, and illustrate the debugging technique with a molecular dynamics simulation. We evaluate the performance of the tool on a 12,000 cores Cray XE6.

Programming languages have advanced tremendously over the years, but program debuggers have hardly changed. Sequential debuggers do little more than allow a user to control the flow of a program and examine its state. Parallel ones support the same operations on multiple processes, which are adequate with a small number of processors, but become unwieldy and ineffective on very large machines. Typical scientific codes have enormous multi-dimensional data structures and it is impractical to expect a user to view the data using traditional display techniques. In this paper we discuss the use of debug-time assertions, and show that these can be used to debug parallel programs. The techniques reduce the debugging complexity because they reason about the state of large arrays without requiring the user to know the expected value of every element. Assertions can be expensive to evaluate, but their performance can be improved by running them in parallel. We demonstrate the system with a case study finding errors in a parallel version of the Shallow Water Equations, and evaluate the performance of the tool on a 4,096 cores Cray XE6.

Debugging parallel programs is an order of magnitude more complex than sequential ones, and yet, most parallel debuggers provide little extra functionality than their sequential counterparts. This problem becomes more serious as computational codes become more complex, involving larger data structures, and as the machines become larger. Peta-scale machines consisting of millions of cores pose a significant challenge for existing techniques. We argue that debugging must become more data-centric, and believe that "assertions" provide a useful model. Assertions allow a user to declare their expectations about the program state as a whole rather than focusing on that of only a single process state. Previously, we have implemented a special type of assertion that supports debugging applications as they evolve or are ported to different platforms. They allow a user to compare the state of one program against another reference version. These 'relative debugging' assertions, whilst powerful, pose significant implementation challenges for large peta-scale machines. In this paper we discuss a hashing technique that provides a scalable solution for very large problems on very large machines. We illustrate the scheme on 65k cores of Kraken, a Cray XT5 at the University of Tennessee.

The purpose of an application performance tool is to help users identify whether or not their application are running efficiently on the computing resources available. However, the scale of current and future high end systems, as well as increasing system software and architecture complexity, brings a new set of challenges for performance tuning. To achieve high performance on peta-scale systems, users need a new infrastructure for performance measurement that can automate the process of performance analysis. In order to address these issues, we designed an automatic performance analysis framework, which contains innovative techniques that use a knowledge base and set of performance models. In this paper we outline the key aspects of this design and present the features from this design that have been implemented in the Cray Performance Analysis Toolset.

Abstract Co‐Array Fortran (CAF) is a parallel programming extension to Fortran that provides a straightforward mechanism for representing distributed memory communication and, in particular, one‐sided communication. Although this integration of communication primitives with the language improves programmer productivity, this new level of abstraction makes the analysis of CAF performance mode difficult. This situation is due, in part, to a lack of tools for the analysis of CAF applications. In this paper, we present an extension to the KOJAK toolkit based on a source‐to‐source translator that supports performance instrumentation, data collection, trace generation, and performance visualization of CAF applications. We illustrate this approach with a performance visualization of a CAF version of the Halo kernel benchmark using the VAMPIR event trace visualization tool. Copyright © 2007 John Wiley & Sons, Ltd.

Application developers are facing new and more complicated performance tuning and optimization problems as architectures become more complex. In order to achieve reasonable performance on these systems, HPC users need help from performance analysis tools. In this tutorial we will introduce the principles of experimental performance instrumentation, measurement, and analysis, with an overview of the major issues, techniques, and resources in performance tools development, as well as an overview of the performance measurement tools available from vendors and research groups. In addition, we will discuss cutting edge issues, such as automatic performance analysis and analysis of grid applications. Our goals are twofold: first, we will provide background information about methods and techniques for performance measurement and analysis, including practical tricks and tips, so that you can exploit available tools effectively and efficiently. Second, you will learn about simple portable techniques for measuring the performance of your parallel application.

Detailed cache simulation can be useful to both system developers and application writers to understand an application’s performance. However, measuring long running programs can be extremely slow. In this paper we present a technique to use dynamic sampling of trace snippets throughout an application’s execution. We demonstrate that our approach improves accuracy compared to sampling a few timesteps at the beginning of execution by judiciously choosing the frequency, as well as the points in the control flow, at which samples are collected. Our approach is validated using the SIGMA tracing and simulation framework for the IBM Power family of processors.

The design and development of a new computer system is a lengthy process, with a considerable amount of time elapsed between the beginning of development and first hardware availability. Hence, fast and reasonably accurate simulation of processor architecture has become critical as an enabling mechanism for software engineers to develop and tune system software and applications. In this paper, we present the time-stamped timing model extensions to the BlueGene/L functional simulator. These extensions were implemented to create a pseudo cycle-accurate simulator capable of providing tracing capabilities for detection of bottlenecks and for performance tuning of applications, before the actual hardware became available. Our validation tests, using the DAXPY kernel and the serial version of the NAS benchmarks, show that our pseudo cycle-accurate simulator provides timing information within 15% of the times measured using the actual BlueGene/L hardware. In addition, we present a couple of case studies, which describes how this simulator can be used for identification of performance bottlenecks and for application tuning.

In this paper we present the design and implementation of RVision (Remote Vision), an open architecture, high configurable tool for cluster monitoring. We focus on the description of it's modular architecture, emphasizing the new concepts we are introducing for cluster monitoring, such as monitoring sessions and the support for dynamic linking of monitoring libraries. In addition, we measure intrusion with several benchmarks and applications under different scenarios. RVision distinguishes itself from other available tools for cluster monitoring because of its open architecture, high configurability, and low intrusion. It is being used in production mode, in our research center, and has proven itself as a powerful alternative for cluster monitoring, especially in heterogeneous clusters and cluster of clusters.

In this paper we present SIGMA (Simulation Infrastructure to Guide Memory Analysis), a new data collection framework and family of cache analysis tools. The SIGMA environment provides detailed cache information by gathering memory reference data using software-based instrumentation. This infrastructure can facilitate quick probing into the factors that influence the performance of an application by highlighting bottleneck scenarios including: excessive cache/TLB misses and inefficient data layouts. The tool can also assist in perturbation analysis to determine performance variations caused by changes to architecture or program. Our validation tests using the SPEC Swim benchmark show that most of the performance metrics obtained with SIGMA are within 1% of the metrics obtained with hardware performance counters, with the advantage that SIGMA provides performance data on a data structure level, as specified by the programmer.

The complexity of both parallel architectures and parallel applications poses several problems for the development of performance analysis and optimization tools. In this paper, we describe the motivations and the main aspects of the design of the Dynamic Probe Class Library (DPCL), an object based C++ class library that provides an infrastructure to reduce the cost of writing instrumentation for performance tools. Additionally, we present some of the performance tools built on top of DPCL, which demonstrate the power and flexibility of the library.

Complex, distributed applications pose new challenges for performance analysis and optimization. This paper outlines an online approach to performance analysis where developers are active participants, using integrated measurement and immersive performance visualization to tune parallel and distributed applications.

In this paper we present the design of SvPablo, a language independent performance analysis and visualization system that can be easily extended to new contexts with minimal changes to the software infrastructure. At present, SvPablo supports analysis of applications written in C, Fortran 77, Fortran 90, and HPF on a variety of sequential and parallel systems. In addition to capturing application data via software instrumentation, SvPablo also exploits hardware performance counters to capture the interaction of software and hardware. Both hardware and software performance data are summarized during program execution, enabling measurement of programs that execute for hours or days on hundreds of processors. This performance data is stored in a format designed to be language transparent and portable. We demonstrate the usefulness of SvPablo for tuning application programs with a case study running on an SGI Origin 2000.

This article describes the main techiques developed for FALCON's MATLAB-to-Fortran 90 compiler. FALCON is a programming environment for the development of high-performance scientific programs. It combines static and dynamic inference methods to translate MATLAB programs into Fortran 90. The static inference is supported with advanced value propagation techniques and symbolic algorithms for subscript analysis. Experiments show that FALCON's MATLAB translator can generate code that performs more than 1000 times faster than the interpreted version of MATLAB and substantially faster than commercially available MATLAB compilers on one processor of an SGI Power Challenge. Futhermore, in most cases we have tested, the compiler-generated code is as fast as corresponding hand-written programs.

In this paper, we describe the inference mechanism used by the FALCON system to translate MATLAB programs to Fortran 90. FALCON is a programming environment for the development of scientific libraries and applications. The objective of the MATLAB compiler is to allow program development to take place in a userfriendly, interactive environment without sacrificing performance. FALCON's inference mechanism combines static and dynamic inference methods for intrinsic type, rank, and shape inference, and is supported by a sophisticated symbolic value propagation algorithm. Experimental results show that FALCON's MATLAB compiler can generate code that is over 1000 times faster than MATLAB on a uniprocessor SGI Power Challenge, and is often as fast as handwritten Fortran programs.

We present the design of the GFDL ocean circulation model as adapted for simulations of the Mediterranean basin for the Cedar multicluster architecture. The model simulates the basic aspects of large-scale, baroclinic ocean circulation, including treatment of irregular bottom topography. The data and computational mapping strategies and their effect on the design are discussed. The code was parameterized to offer several choices for data partitionings of the computational domain, for placement strategies for the data in the memory hierarchy, and for the number of clusters and processors used in the computational hierarchy of Cedar. The experiments and performance trends are discussed. Using four clusters and 32 processors the code demonstrates significant speedup compared to a single cluster and compared to a single processor. 1 Introduction The numerical modeling of ocean circulation is a task of great importance, both in its own right, as well as a component of climate studies. Num...