Modern scientific instruments are becoming essential for discoveries because they provide unprecedented insight into physical or biological events – often in real time. However, these instruments may generate large amounts of data, and increasingly they require sophisticated e-infrastructure for analysis, storage and archive. The increasing complexity and scale of the data, processing steps and systems has made it difficult for domain scientists to perform their research, narrowing the user base to a select few. In this paper, we present a framework that democratises large-scale instrument-based science, increasing the number of researchers who can engage. We discuss a prototype at the University of Queensland. The system is illustrated through two case studies, one involving light microscopy imaging of the innate immune system, and the other electron microscopy imaging of the SARS-CoV-2 viral proteins.

The increasing amount of data being collected from simulations, instruments and sensors creates challenges for existing e-Science infrastructure. In particular, it requires new ways of storing, distributing and processing data in order to cope with both the volume and velocity of the data. The University of Queensland has recently designed and deployed MeDiCI, a data fabric that spans the metropolitan area and provides seamless access to data regardless of where it is created, manipulated and archived. MeDiCI is novel in that it exploits temporal and spatial locality to move data on demand in an automated manner. This means that data only needs to reside locally in high speed storage whilst being manipulated, and it can be archived transparently in high capacity, but slower, technologies at other times. MeDiCI is built on commercially available technologies. In this paper, we describe these innovations and present some early results.

Relative debugging traces software errors by comparing two executions of a program concurrently - one code being a reference version and the other faulty. Relative debugging is particularly effective when code is migrated from one platform to another, and this is of significant interest for hybrid computer architectures containing CPUs accelerators or coprocessors. In this paper we extend relative debugging to support porting stencil computation on a hybrid computer. We describe a generic data model that allows programmers to examine the global state across different types of applications, including MPI/OpenMP, MPI/OpenACC, and UPC programs. We present case studies using a hybrid version of the `stellarator' particle simulation DELTA5D, on Titan at ORNL, and the UPC version of Shallow Water Equations on Crystal, an internal supercomputer of Cray. These case studies used up to 5,120 GPUs and 32,768 CPU cores to illustrate that the debugger is effective and practical.

Summary Contemporary parallel debuggers allow users to control more than one processing thread while supporting the same examination and visualisation operations of that of sequential debuggers. This approach restricts the use of parallel debuggers when it comes to large scale scientific applications run across hundreds of thousands compute cores. First, manually observing the runtime data to detect error becomes impractical because the data is too big. Second, performing expensive but useful debugging operations becomes infeasible as the computational codes become more complex, involving larger data structures, and as the machines become larger. This study explores the idea of a data‐centric debugging approach, which could be used to make parallel debuggers more powerful. It discusses the use ofad hocdebug‐time assertions that allow a user to reason about the state of a parallel computation. These assertions support the verification and validation of program state at runtime as a whole rather than focusing on that of only a single process state. Furthermore, the debugger's performance can be improved by exploiting the underlying parallel platform because the available compute cores can execute parallel debugging functions, while a program is idling at a breakpoint. We demonstrate the system with several case studies and evaluate the performance of the tool on a 20 000 cores Cray XE6. Copyright © 2013 John Wiley & Sons, Ltd.

Detecting and isolating bugs that arise only at high processor counts is a challenging task. Over a number of years, we have implemented a special debugging method, called "relative debugging," that supports debugging applications as they evolve or are ported to larger machines. It allows a user to compare the state of a suspect program against another reference version even as the number of processors is increased. The innovative idea is the comparison of runtime data to reason about the state of the suspect program. While powerful, a naïve implementation of the comparison phase does not scale to large problems running on large machines. In this paper, we propose two different solutions including a hash-based scheme and a direct point-to-point scheme. We demonstrate the implementation, a case study, as well as the performance, of our techniques on 20K cores of a Cray XE6 system.

Parallel debugging faces challenges in both scalability and efficiency. A number of advanced methods have been invented to improve the efficiency of parallel debugging. As the scale of system increases, these methods highly rely on a scalable communication protocol in order to be utilized in large-scale distributed environments. This paper describes a debugging middleware that provides fundamental debugging functions supporting multiple communication protocols. Its pluggable architecture allows users to select proper communication protocols as plug-ins for debugging on different platforms. It aims to be utilized by various advanced debugging technologies across different computing platforms. The performance of this debugging middleware is examined on a Cray XE Supercomputer with 21,760 CPU cores.

Traditional debuggers are of limited value for modern scientific codes that manipulate large complex data structures. This paper discusses a novel debug-time assertion, called a "Statistical Assertion", that allows a user to reason about large data structures, and the primitives are parallelised to provide an efficient solution. We present the design and implementation of statistical assertions, and illustrate the debugging technique with a molecular dynamics simulation. We evaluate the performance of the tool on a 12,000 cores Cray XE6.

Programming languages have advanced tremendously over the years, but program debuggers have hardly changed. Sequential debuggers do little more than allow a user to control the flow of a program and examine its state. Parallel ones support the same operations on multiple processes, which are adequate with a small number of processors, but become unwieldy and ineffective on very large machines. Typical scientific codes have enormous multi-dimensional data structures and it is impractical to expect a user to view the data using traditional display techniques. In this paper we discuss the use of debug-time assertions, and show that these can be used to debug parallel programs. The techniques reduce the debugging complexity because they reason about the state of large arrays without requiring the user to know the expected value of every element. Assertions can be expensive to evaluate, but their performance can be improved by running them in parallel. We demonstrate the system with a case study finding errors in a parallel version of the Shallow Water Equations, and evaluate the performance of the tool on a 4,096 cores Cray XE6.

In this paper, we present a framework that enables scientists to steer computations executing over large-scale grid computing environments. By using computational steering, users can dynamically control their simulations or computations to reach expected results more efficiently. The framework supports steerable applications by introducing an asynchronous iterative MapReduce programming model that is deployed using Hadoop over a set of virtual machines executing on a multi-cluster grid. To tolerate the heterogeneity between different sites, results are collected asynchronously and users can dynamically interact with their computations to adjust the area of interest. According to users dynamic interaction, the framework can redistribute the computational overload between the heterogeneous sites and explore the user's interest area by using more powerful sites when possible. With our framework, the bottleneck induced by synchronisation between different sites is considerably avoided, and therefore the response to users interaction is satisfied more efficiently. We illustrate and evaluate this framework with a scientific application that aims to t models of the Milky Way galaxy structure to stars observed by the Sloan Digital Sky Survey.

To ensure Quality of Service (QoS) for data centers, it is critical to enforce a fair share of storage resources between competing users. Interposed schedulers are one of the most practical methods for performance isolation. Most fair queuing-based proportional sharing algorithms for existing interposed scheduler are variants of counterparts designed for network routers and may result in breaking the fairness of proportional sharing required by Service Level Agreements for storage systems. This paper presents a novel algorithm to address this problem. As an extension of the fair queuing-based algorithm, it can dynamically adapt to the performance variation of storage systems and guarantee a fair sharing of resources as well as satisfying the minimal performance requirements for different clients. The design and performance evaluation are presented.

The MapReduce programming model allows users to easily develop distributed applications in data centers. However, many applications cannot be exactly expressed with MapReduce due to their specific characteristics. For instance, genetic algorithms (GAs) naturally fit into an iterative style. That does not follow the two phase pattern of MapReduce. This paper presents an extension to the MapReduce model featuring a hierarchical reduction phase. This model is called MRPGA (MapReduce for parallel GAs), which can automatically parallelize GAs. We describe the design and implementation of the extended MapReduce model on a .NET-based enterprise grid system in detail. The evaluation of this model with its runtime system is presented using example applications.

The Gridbus workflow management system (GWMS) is designed to execute scientific applications, expressed in the form of workflows, onto Grid and Cloud resources. With the help of this system, we demonstrate a computational and data-intensive Image Registration (IR) workflow for functional Magnetic Resonance Imaging (fMRI) applications on Clouds and global Grids. We also present a demonstration of the Aneka System. Aneka is a .NET based Cloud software platform that provides: (i) a configurable service container hosting pluggable services for discovering, scheduling various types of workloads and (ii) a flexible and extensible framework supporting various programming models. We use distributed Grid and Cloud resources from Australia, Austria, France, Japan, and USA for the executions of image registration workflow, and resources from Melbourne University for executing map-reduce applications in Aneka cloud environment.

In this paper, we present the design of Aneka, a .NET based service-oriented platform for desktop grid computing that provides: (i) a configurable service container hosting pluggable services for discovering, scheduling and balancing various types of workloads and (ii) a flexible and extensible framework/API supporting various programming models including threading, batch processing, MPI and dataflow. Users and developers can easily use different programming models and the services provided by the container to run their applications over desktop Grids managed by Aneka. We present the implementation of both the essential and advanced services within the platform. We evaluate the system with applications using the grid task and dataflow models on top of the infrastructure and conclude with some future directions of the current system.