The remote visual exploration of live data generated by scientific simulations is useful for scientific discovery, performance monitoring, and online validation for the simulation results. Online visualization methods are challenged, however, by the continued growth in the volume of simulation output data that has to be transferred from its source - the simulation running on the high end machine - to where it is analyzed, visualized, and displayed. A specific challenge in this context is limits in the communication bandwidth between data source(s) and sinks. Previous work places queries `near' data sources, exploiting their data reduction capabilities, but such work does not address the common scenario in which scientists make multiple different queries on the data being produced. This paper considers the general case in which science users are interested in different (sub)sets of the data produced by a high end simulation. We offer the FlexQuery online data query system that can deploy and execute data queries `along' the I/O and analytics pipelines. FlexQuery carefully extends such analytics pipelines, using online performance monitoring and data location tracking, to realize data queries in ways that minimize additional data movement and offer low latency in data query execution. Using a real-world scientific application - the Maya astrophysics code and its analytics workflow - we demonstrate FlexQuery's ability to dynamically deploy queries for low-latency remote data visualization.

Our on-going project, Unibus, aims to facilitate provisioning and aggregation of multifaceted resources from resource providers and end-users' perspectives. To achieve that, Unibus proposes (1) the Capability Model and mediators (resource drivers) to virtualize access to diverse resources, and (2) soft and successive conditioning to enable automatic and user-transparent resource provisioning. In this paper we examine the Unibus concepts and prototype in a real situation of aggregation of two commercial clouds and execution of benchmarks on aggregated resources. We also present and discuss benchmarks' results.

High-end machines at modern HPC centers are constantly undergoing hardware and system software upgrades - necessitating frequent rebuilds of application codes. The number of possible combinations of compilers, libraries, application build configurations, differing hardware architectures, etc, makes the process of building applications very onerous, requiring expert build knowledge from different domains. Our ongoing Harness Workbench Toolkit (HWT) project aims to foster and streamline the entire build process on heterogeneous computational platforms. This paper focuses on a key research issue of the HWT that regards facilitating and enhancement portability of build systems across multifarious machines, with particular respect to scientific software commonly used in the HPC community. The article presents a novel HWT approach based on the concept of generic build systems and profiles which encapsulate build knowledge provided independently by relevant experts. The paper describes profiles, the logistics of storing and retrieving build information, and interfacing to user-guided builds. We also report on experiences with applying the HWT approach to two scientific production codes (CPMD, GAMESS) on Cray XT4.

Fast hardware turnover in supercomputing centers, stimulated by rapid technological progress, results in high heterogeneity among HPC platforms, and necessitates that applications are ported and adapted frequently. The cutting-edge nature of the hardware mandates customized performance tuning, which, coupled with continuously growing application complexity, makes the process inherently and increasingly challenging. In this paper, we analyze build procedures of a representative set of HPC applications, and attempt to identify commonalities that can be exploited to enhance cross-platform portability. We then propose a novel method for reducing non-portabilities while preserving high performance. The approach, based on profiles that capture and isolate non-portable features at various levels, requires only a moderate amount of changes to existing makefiles. It leverages the expertise of system designers and administrators, and reduces burdens placed on application scientists. As a proof of concept, we discuss the application of our methodology to enhancing portability of the Mile application across heterogeneous HPC platforms.

Heterogeneous distributed computing platforms continue to require substantial effort in terms of software installation and management for efficient use, often necessitating manual intervention by resource providers and end-users. In this paper we propose a novel model of resource sharing that is a viable alternative to that commonly adopted in the grid community. Our model shifts the resource virtualization and aggregation responsibilities to software at the client side, taking these burdens away from resource providers. Following this model, we have designed the Zero-Force MPI toolkit that automates installation, run, and post-processing stages of HPC applications, thus allowing application scientists to focus on using computational resources instead of managing them. Through the provided console, MPI can be soft-installed across a set of distributed resources with just a few commands. The prototype implementation of the ZF-MPI toolkit is based on FTMPI. To evaluate the toolkit, we conducted experiments with deploying and executing the NAS Parallel Benchmarks. Results demonstrate that the run-not-install approach exemplified by ZF-MPI is effective and may substantially increase productivity of application execution.

Shared HPC platforms continue to require substantial effort for software installation and management, often necessitating manual intervention and tedious procedures. We propose a novel model of resource sharing that shifts resource virtualization and aggregation responsibilities to client-side software, thus reducing the burdens on resource providers.The Zero-Force MPI toolkit automates the installation, build, run, and post-processing stages of HPC applications, thus allowing application scientists to focus on using resources instead of managing them. Through a provided console, MPI runtime systems, support libraries, application executables, and needed datafiles can be soft-installed across distributed resources with just a few commands. Built-in data synchronization capabilities simplify common HPC development tasks, saving end-user time and effort. To evaluate ZF-MPI, we conducted experiments with the NAS Parallel Benchmarks. Results demonstrate that the proposed run-not-install approach is effective and may substantially increase overall productivity.