To perform computational experiments at greater scale and in less time, enterprises are increasingly looking to dynamically expand their computing capabilities through the temporary addition of cloud resources (aka "cloudbursting"). Computational infrastructure can be dismantled in minutes with no long-term capital investments. However, research is needed to identify which properties of an application best determine the potential benefits of cloudbursting. For example, there are certainly situations where the cost to transfer the necessary input data from the enterprise to the cloud (to execute the application in the cloud) outweighs the value of simply waiting until resources become available in-house. To better understand and quantify these general issues, we perform a concrete analysis of the value of cloudbursting for a large-scale application we have previously created to process and derive environmental results from satellite imagery. More specifically, we compare three versions of the application (an all-cloud design, a version that runs in-house on our cluster, and a hybrid cloudbursting version) on dimensions of debug ability, fault tolerance, correctness, economics, usability, and run-time speed. We find that for our application, cloudbursting is effective primarily because we were able to design the application so that its I/O behavior does not preclude remote (cloud) execution, we were able to minimize developmental cost by constructing a cloud run-time environment that is very similar to our in-house environment, and we achieve good run-time performance in our cloud-based executions (for example, we describe how a representative computation that takes 2 ½ hours in-house is completed in 35 minutes via cloudbursting). By generalizing this analysis, we believe that we contribute guidance to the broader community on the value of cloudbursting for escience applications.

Large scale scientific data transfers are central to scientific processes. Data from large experimental facilities have to be moved to local institutions for analysis or often data needs to be moved between local clusters and large supercomputing centers. In this paper, we propose and evaluate a network overlay architecture to enable high-throughput, on-demand, coordinated data transfers over wide-area networks. Our work leverages Phoebus and On-demand Secure Circuits and Advance Reservation System (OSCARS) to provide high performance wide-area network connections. OSCARS enables dynamic provisioning of network paths with guaranteed bandwidth and Phoebus enables the coordination and effective utilization of the OSCARS network paths. Our evaluation shows that this approach leads to improved end-to-end data transfer throughput with minimal overheads. The achieved throughput using our overlay was limited only by the ability of the end hosts to sink the data.

Cloud computing has evolved in the commercial space to support highly asynchronous web 2.0 applications. Scientific computing has traditionally been supported by centralized federally funded supercomputing centers and grid resources with a focus on bulk-synchronous compute and data-intensive applications. The scientific computing community has shown increasing interest in exploring cloud computing to serve e-Science applications, with the idea of taking advantage of some of its features such as customizable environments and on-demand resources. Magellan, a recently funded cloud computing project is investigating how cloud computing can serve the needs of mid-range computing and future data-intensive scientific workloads. This paper summarizes the application requirements and business model needed to support the requirements of both existing and emerging science applications, as learned from the early experiences on Magellan and commercial cloud environments. We provide an overview of the capabilities of leading cloud offerings and identify the existent gaps and challenges. Finally, we discuss how the existing cloud software stack may be evolved to better meet e-Science needs, along with the implications for resource providers and middleware developers.

Cloud computing has seen tremendous growth, particularly for commercial web applications. The on-demand, pay-as-you-go model creates a flexible and cost-effective means to access compute resources. For these reasons, the scientific computing community has shown increasing interest in exploring cloud computing. However, the underlying implementation and performance of clouds are very different from those at traditional supercomputing centers. It is therefore critical to evaluate the performance of HPC applications in today's cloud environments to understand the tradeoffs inherent in migrating to the cloud. This work represents the most comprehensive evaluation to date comparing conventional HPC platforms to Amazon EC2, using real applications representative of the workload at a typical supercomputing center. Overall results indicate that EC2 is six times slower than a typical mid-range Linux cluster, and twenty times slower than a modern HPC system. The interconnect on the EC2 cloud platform severely limits performance and causes significant variability.

It can be natural to believe that many of the traditional issues of scale have been eliminated or at least greatly reduced via cloud computing. That is, if one can create a seemingly well functioning cloud application that operates correctly on small or moderate-sized problems, then the very nature of cloud programming abstractions means that the same application will run as well on potentially significantly larger problems. In this paper, we present our experiences taking MODISAzure, our satellite data processing system built on the Windows Azure cloud computing platform, from the proof-of-concept stage to a point of being able to run on significantly larger problem sizes (e.g., from national-scale data sizes to global-scale data sizes). To our knowledge, this is the longest-running eScience application on the nascent Windows Azure platform. We found that while many infrastructure-level issues were thankfully masked from us by the cloud infrastructure, it was valuable to design additional redundancy and fault-tolerance capabilities such as transparent idempotent task retry and logging to support debugging of user code encountering unanticipated data issues. Further, we found that using a commercial cloud means anticipating inconsistent performance and black-box behavior of virtualized compute instances, as well as leveraging changing platform capabilities over time. We believe that the experiences presented in this paper can help future eScience cloud application developers on Windows Azure and other commercial cloud providers.

Today, our picture of the Universe radically differs from that of just over a decade ago. We now know that the Universe is not only expanding as Hubble discovered in 1929, but that the rate of expansion is accelerating, propelled by mysterious new physics dubbed "Dark Energy." This revolutionary discovery was made by comparing the brightness of nearby Type Ia supernovae (which exploded in the past billion years) to that of much more distant ones (from up to seven billion years ago). The reliability of this comparison hinges upon a very detailed understanding of the physics of the nearby events. As part of its effort to further this understanding, the Nearby Supernova Factory (SNfactory) relies upon a complex pipeline of serial processes that execute various image processing algorithms in parallel on ~10TBs of data.

The combination of low-cost sensors, low-cost commodity computing, and the Internet is enabling a new era of data-intensive science. The dramatic increase in this data availability has created a new challenge for scientists: how to process the data. Scientists today are envisioning scientific computations on large scale data but are having difficulty designing software architectures to accommodate the large volume of the often heterogeneous and inconsistent data. In this paper, we introduce a particular instance of this challenge, and present our design and implementation of a MODIS satellite data reprojection and reduction pipeline in the Windows Azure cloud computing platform. This cloud-based pipeline is designed with a goal of hiding data complexities and subsequent data processing and transformation from end users. This pipeline is highly flexible and extensible to accommodate different science data processing tasks, and can be dynamically scaled to fulfill scientists' various computational requirements in a cost-efficient way. Experiments show that by running a practical large-scale science data processing job in the pipeline using 150 moderately-sized Azure virtual machine instances, we were able to produce analytical results in nearly 90× less time than was possible with a high-end desktop machine. To our knowledge, this is one of the first eScience applications to use the Windows Azure platform.

Abstract Carbon‐climate, like other environmental sciences, has been changing. Large‐scale synthesis studies are becoming more common. These synthesis studies are often conducted by science teams that are geographically distributed and on data sets that are global in scale. A broad array of collaboration and data analytics tools are now available that could support these science teams. However, building tools that scientists actually use is difficult. Also, moving scientists from an informal collaboration structure to one mediated by technology often exposes inconsistencies in the understanding of the rules of engagement between collaborators. We have developed a scientific collaboration portal, calledfluxdata.org, which serves the community of scientists providing and analyzing the global FLUXNET carbon‐flux synthesis data set. The key things we learned or re‐learned during our portal development include: minimize the barrier to entry, provide features on a just‐in‐time basis, development of requirements is an on‐going process, provide incentives to change leaders and leverage the opportunity they represent, automate as much as possible, and you can only learn how to make it better if people depend on it enough to give you feedback. In addition, we also learned that splitting the portal roles between scientists and computer scientists improved user adoption and trust. The fluxdata.org portal has now been in operation for ∼2 years and has become central to the FLUXNET synthesis efforts. Published in 2010 by John Wiley & Sons, Ltd.

The Web services resource framework defines conventions for managing state in distributed systems based on Web services, and WS-notification defines topic-based publish/subscribe mechanisms. We analyze five independent and quite different implementations of these specifications from the perspectives of architecture, functionality, standards compliance, performance, and interoperability. We identify both commonalities among the different systems (e.g., similar dispatching and SOAP processing mechanisms) and differences (e.g., security, programming models, and performance). Our results provide insights into effective implementation approaches. Our results may also provide application developers, system architects, and deployers with guidance in identifying the right implementation for their requirements and in determining how best to use that implementation and what to expect with regard to performance and interoperability.

Securing a Grid environment presents a distinctive set of challenges. This work groups the activities that need to be secured into four categories: naming and authentication; secure communication; trust, policy, and authorization; and enforcement of access control. It examines the current state of the art in securing these activities and introduces new technologies that promise to meet the security requirements of Grids more completely.

Developers and users of high-performance distributed systems often observe performance problems such as unexpectedly low throughput or high latency. Determining the source of the performance problems requires detailed end-to-end instrumentation of all components, including the applications, operating systems, hosts, and networks. However, one must be very careful to design the instrumentation to have extremely low overhead, and not affect the system being monitored. In this paper we present a very light-weight instrumentation system that can be dynamically activated to unobtrusively collect and aggregate detailed end-to-end monitoring information from distributed applications. We also show how emerging "web services" can be used to facilitate remote interaction with this system.

Abstract Developing high‐performance, problem‐solving environments/applications that allow scientists to easily harness the power of the emerging national‐scale ‘Grid’ infrastructure is currently a difficult task. Although many of the necessary low‐level services, e.g. security, resource discovery, remote access to computation/data resource, etc., are available, it can be a challenge to rapidly integrate them into a new application. To address this difficulty we have begun the development of a Python‐based high‐level interface to the Grid services provided by the Globus Toolkit. In this paper we will explain why rapid application development using Grid services is important, look briefly at a motivating example, and finally look at the design and implementation of the pyGlobus package. Copyright © 2002 John Wiley & Sons, Ltd.

Large-scale science and engineering is frequently done through the interaction of collaborating groups, heterogeneous computing resources, information systems, and instruments, all of which are geographically and organizationally dispersed. The overall motivation for "Grids" is to enable the routine interactions of these resources to enhance this type of large-scale science and engineering, and thus substantially increase the computing and data handling capabilities available to science and engineering projects. However, even if this environment works in every other way, it will not be viable if it is constantly disrupted by hackers and their kin. Distributed applications are potentially more vulnerable than conventional scientific problem solving environments because there are substantially more targets to attack in order to impact a single application. Much of the overall security of Grids is inherited from the security of the underlying systems. There are, however, some security considerations at the Grid level that are independent of the underlying systems, and we focus on this latter aspect.