Cloud computing is emerging as a promising alternative to supercomputers for some high-performance computing (HPC) applications. With cloud as an additional deployment option, HPC users and providers are faced with the challenges of dealing with highly heterogeneous resources, where the variability spans across a wide range of processor configurations, interconnects, virtualization environments, and pricing models. In this paper, we take a holistic viewpoint to answer the question-why and whoshould choose cloud for HPC, for what applications, and how should cloud be used for HPC? To this end, we perform comprehensive performance and cost evaluation and analysis of running a set of HPC applications on a range of platforms, varying from supercomputers to clouds. Further, we improve performance of HPC applications in cloud by optimizing HPC applications' characteristics for cloud and cloud virtualization mechanisms for HPC. Finally, we present novel heuristics for online application-aware job scheduling in multi-platform environments. Experimental results and simulations using CloudSim show that current clouds cannot substitute supercomputers but can effectively complement them. Significant improvement in average turnaround time (up to 2X)and throughput (up to 6X) can be attained using our intelligent application-aware dynamic scheduling heuristics compared tosingle-platform or application-agnostic scheduling.

Cloud computing is emerging as an alternative to supercomputers for some of the high-performance computing (HPC) applications that do not require a fully dedicated machine. With cloud as an additional deployment option, HPC users are faced with the challenges of dealing with highly heterogeneous resources, where the variability spans across a wide range of processor configurations, interconnections, virtualization environments, and pricing rates and models. In this paper, we take a holistic viewpoint to answer the question - why and who should choose cloud for HPC, for what applications, and how should cloud be used for HPC? To this end, we perform a comprehensive performance evaluation and analysis of a set of benchmarks and complex HPC applications on a range of platforms, varying from supercomputers to clouds. Further, we demonstrate HPC performance improvements in cloud using alternative lightweight virtualization mechanisms - thin VMs and OS-level containers, and hyper visor- and application-level CPU affinity. Next, we analyze the economic aspects and business models for HPC in clouds. We believe that is an important area that has not been sufficiently addressed by past research. Overall results indicate that current public clouds are cost-effective only at small scale for the chosen HPC applications, when considered in isolation, but can complement supercomputers using business models such as cloud burst and application-aware mapping.

Version control systems are an indispensable part of the software engineering process. Without them, large software development projects would soon incur in loss of productivity and severe maintenance issues. The revision history is invaluable during reviewing process as well as bug hunting. With the advent of Web 2.0, many tasks have been migrated to a cloud environment, and among them is software development. This spikes the need for a fast, always-on version control system residing in the cloud. Moreover, the cloud computational power can be leveraged to provide a higher degree of automation. In this paper, we describe Protego, a cloud-based version control system targeting flexibility and automation. Flexibility in the way projects and histories are recorded and updated, automation in the way users interact with the system, with a reduced need for everyday tedious operations.

To deliver scalability for business processes running in multi-tenant cloud environment has become a critical goal for enterprise business service providers. Many such software systems are composed of data intensive operations, offered via the Internet as services, and they often perform collaborative analysis of data. Collaborative analysis of data imposes new requirements of scalability on the underneath platform. We need to approach a supporting platform for multiple systems in a multi-tenant cloud environment to achieve better performance. We present our solution of scaling-out for inter-tenant data flows between collaborative data-intensive cloud systems deployed with different VMs. In this paper, we define an abstraction of “shared data flows” to capture data co-analysis and dependency among services of such systems. We describe a model based on the abstraction that is designed with elements to address the challenges in supporting multi-tenant environment to achieve an improved scalability. Our objective is to provide optimization opportunities on bandwidth and throughput of the collaborative service systems of various tenants.

This paper presents a scheme to optimize the mapping of HPC applications to a set of hybrid dedicated and cloud resources. First, we characterize application performance on dedicated clusters and cloud to obtain application signatures. Then, we propose an algorithm to match these signatures to resources such that performance is maximized and cost is minimized. Finally, we show simulation results revealing that in a concrete scenario our proposed scheme reduces the cost by 60% at only 10-15% performance penalty vs. a non optimized configuration. We also find that the execution overhead in cloud can be minimized to a negligible level using thin hypervisors or OS-level containers.

This paper focuses on the use of GPGPU-based clusters for hierarchical N-body simulations. Whereas the behavior of these hierarchical methods has been studied in the past on CPU-based architectures, we investigate key performance issues in the context of clusters of GPUs. These include kernel organization and efficiency, the balance between tree traversal and force computation work, grain size selection through the tuning of offloaded work request sizes, and the reduction of sequential bottlenecks. The effects of various application parameters are studied and experiments done to quantify gains in performance. Our studies are carried out in the context of a production-quality parallel cosmological simulator called ChaNGa. We highlight the re-engineering of the application to make it more suitable for GPU-based environments. Finally, we present performance results from experiments on the NCSA Lincoln GPU cluster, including a note on GPU use in multistepped simulations.

Parallel applications typically run in batch mode, sometimes after long waits in a scheduler queue. In some situations, it would be desirable to interactively add new functionality to the running application, without having to recompile and rerun it. For example, a debugger could upload code to perform consistency checks, or a data analyst could upload code to perform new statistical tests. This paper presents a scalable technique to dynamically insert code into running parallel applications. We describe and evaluate an implementation of this idea that allows a user to upload Python code into running parallel applications. This uploaded code will run in concert with the main code. We prove the effectiveness of this technique in two case studies: parallel debugging to support introspection and data analysis of large cosmological datasets.

Cosmological simulators are an important component in the study of the formation of galaxies and large scale structures, and can help answer many important questions about the Universe. Despite their utility, existing parallel simulators do not scale effectively on modern machines containing thousands of processors. In this paper we present ChaNGa, a production simulator based on the CHARM++ infrastructure. To achieve scalable performance, ChaNGa employs various optimizations that maximize the overlap between computation and communication. We present experimental results of ChaNGa simulations on machines with thousands of processors, including the IBM Blue Gene/L and the Cray XT3. The paper goes on to highlight efforts toward even more efficient and scalable cosmological simulations. In particular, novel load balancing schemes that base decisions on certain characteristics of tree-based particle codes are discussed. Further, the multistepping capabilities of ChaNGa are presented, as are solutions to the load imbalance that such multiphase simulations face. We outline key requirements for an effective practical implementation and conclude by discussing preliminary results from simulations run with our multiphase load balancer.

Simulations of astrophysical problems capable of answering questions like "How did the Milky Way form?" or "What is the origin of the spin of terrestrial planets?" require weeks of processing with sustained Teraflop, or even Petaflop, performance. This can be achieved only by utilizing supercomputers with thousands of processors. To meet this challenge, we have developed ChaNGa (former ParallelGravity), a parallel simulator for cosmological interaction based on the Charm++ parallel framework.ChaNGa is one of the first cosmological simulators to attempt scaling to thousands of processors. The approach used, based on parallelization via Charm++, has enabled unprecedented performance results on current large machines. We have scaled relatively small-sized simulations, which are the hardest ones to parallelize, to thousands of processors on various platforms. We will show the infrastructure of both the Charm++ framework and the ChaNGa simulator, together with the key features to scale to large configurations and corresponding results.