Data enters are at the core of a wide variety of daily ICT utilities, ranging from scientific computing to online gaming. Due to the scale of today's data enters, the failure of computing resources is a common occurrence that may disrupt the availability of ICT services, leading to revenue loss. Although many high availability (HA) techniques have been proposed to mask resource failures, datacenter users' -- who rent datacenter resources and use them to provide ICT utilities to a global population' -- still have limited management options for dynamically selecting and configuring HA techniques. In this work, we propose Availability-on-Demand (AoD), a mechanism consisting of an API that allows datacenter users to specify availability requirements which can dynamically change, and an availability-aware scheduler that dynamically manages computing resources based on user-specified requirements. The mechanism operates at the level of individual service instance, thus enabling fine-grained control of availability, for example during sudden requirement changes and periodic operations. Through realistic, trace-based simulations, we show that the AoD mechanism can achieve high availability with low cost. The AoD approach consumes about the same CPU hours but with higher availability than approaches which use HA techniques randomly. Moreover, comparing to an ideal approach which has perfect predictions about failures, it consumes 13% to 31% more CPU hours but achieves similar availability for critical parts of applications.

The elasticity promised by cloud computing does not come for free. Providers need to reserve resources to allow users to scale on demand, and cope with workload variations, which results in low utilization. The current response to this low utilization is to re-sell unused resources with no Service Level Objectives (SLOs) for availability. In this paper, we show how to make some of these reclaimable resources more valuable by providing strong, long-term availability SLOs for them. These SLOs are based on forecasts of how many resources will remain unused during multi-month periods, so users can do capacity planning for their long-running services. By using confidence levels for the predictions, we give service providers control over the risk of violating the availability SLOs, and allow them trade increased risk for more resources to make available. We evaluated our approach using 45 months of workload data from 6 production clusters at Google, and show that 6--17% of the resources can be re-offered with a long-term availability of 98.9% or better. A conservative analysis shows that doing so may increase the profitability of selling reclaimed resources by 22--60%.

A new era of Cloud Computing has emerged, but the characteristics of Cloud load in data centers is not perfectly clear. Yet this characterization is critical for the design of novel Cloud job and resource management systems. In this paper, we comprehensively characterize the job/task load and host load in a real-world production data center at Google Inc. We use a detailed trace of over 25 million tasks across over 12,500 hosts. We study the differences between a Google data center and other Grid/HPC systems, from the perspective of both work load (w.r.t. jobs and tasks) and host load (w.r.t. machines). In particular, we study the job length, job submission frequency, and the resource utilization of jobs in the different systems, and also investigate valuable statistics of machine's maximum load, queue state and relative usage levels, with different job priorities and resource attributes. We find that the Google data center exhibits finer resource allocation with respect to CPU and memory than that of Grid/HPC systems. Google jobs are always submitted with much higher frequency and they are much shorter than Grid jobs. As such, Google host load exhibits higher variance and noise.

Prediction of host load in Cloud systems is critical for achieving service-level agreements. However, accurate prediction of host load in Clouds is extremely challenging because it fluctuates drastically at small timescales. We design a prediction method based on Bayes model to predict the mean load over a long-term time interval, as well as the mean load in consecutive future time intervals. We identify novel predictive features of host load that capture the expectation, predictability, trends and patterns of host load. We also determine the most effective combinations of these features for prediction. We evaluate our method using a detailed one-month trace of a Google data center with thousands of machines. Experiments show that the Bayes method achieves high accuracy with a mean squared error of 0.0014. Moreover, the Bayes method improves the load prediction accuracy by 5.6 -- 50% compared to other state-of-the-art methods based on moving averages, auto-regression, and/or noise filters.

It is with great pleasure that we introduce you to this special issue with the best work presented in CCGrid'2007, the 7th IEEE International Symposium on Cluster Computing and the Grid, held in Rio de Janeiro, Brazil, on May 14-17, 2007.Over the years, CCGrid has become a premium conference of truly international coverage, bringing together researchers and practitioners, and enabling them to share their insight, results, and experience in the multi-faceted areas of Grid and cluster computing.Overall, 330 people attended the event, a CCGrid all-time record.This is a testimony to the fact that the grid and cluster computing communities continue to grow, and many early projects that were started in the beginning of the decade (when the conference series began) are now maturing and producing significant results.The international coverage and impact of CCGrid are also worth highlighting.After Australia, Germany, Japan, the United States of America, the United Kingdom, and Singapore, CCGrid comes to Brazil without yet repeating a host country.Similarly, these proceedings carry papers from 22 different countries: Algeria,

This paper discusses an approach to avoid incorrect results in the execution of automatic tests of multi-threaded systems. We argue that such incorrect results have two main sources. First, it is typically difficult to determine when all threads have finished processing and thus when it is safe to perform the test assertions. Second, background threads can change the system state while assertions are being performed, thus producing non-deterministic results. The main contributions of this work are: (i) a generic approach that ensures that test assertions are performed in a safe moment; (ii) implementation details of such an approach using aspect-oriented programming (AOP); and (Hi) an evaluation of the proposed approach.

BitTorrent is currently a de facto standard for scalable content-distribution. However, its peer-to-peer model for resource allocation does not provide high availability and its performance depends on best-effort contributions given by peers. This has motivated several content-providers to use a hybrid model in which they operate a superpeer in order to attain a higher quality of service. In this paper, we use BitTorrent traces and analytical modelling to investigate the cost incurred by such an entity in relation to the benefits it can provide to the system.

The content-based image retrieval (CBIR) has great interest of the medical community, because it is capable of retrieval similar images stored in servers that have known pathologies. However, an efficient and reliable CBIR solution has not been achieved yet, due to the complexity of the medical image and the great volume they represent. This work proposes a new methodology based on higher processing provided by Grid Computing technology to achieve the CBIR using the registration algorithms. The registration procedure use two metrics, square difference metric (SDM) and cross correlation (CC). Both metrics showed higher efficiency, SDM obtained precision average of 0.83% (breast image) and 0.94% (head image), the CC showed precision of 0.81% (breast) and 0.52% (head). The higher computational cost related to the registration algorithms was amortized by Grid Computing, that was capable of ensure data secure and represent a low cost solution to small clinics and public hospitals. Grid technologies open new opportunities to investigate the contribution on applying the registration algorithms to CBIR and new advances are expected.

Abstract Here we present and evaluate relative accounting, an autonomous accounting scheme that provides accurate results even when the parties (consumer and provider) do not trust each other. Relative accounting relies on the observed relative performance amongst the parties. As such, the basic requirement to use it is that resource consumers must also be resource providers. Relative accounting is totally autonomous in the sense that it uses only local information, i.e. there is no exchange of information between the parties. This allows for the deployment of the autonomous accounting without requiring any sort of identification infrastructure, such as certificate authorities. Not requiring trust or sophisticated infrastructure makes relative accounting a perfect fit for peer‐to‐peer Grids, which aim to scale much further than traditional Grids by allowing free unidentified entry into the Grid. Our results show that relative accounting performs very close to a perfect accounting, whose implementation is infeasible in most systems, including those we target. Relative accounting was developed to work with OurGrid, a peer‐to‐peer Grid in production since December 2004, but it can also be used in other peer‐to‐peer Grids. Copyright © 2006 John Wiley & Sons, Ltd.

Abstract Good water management is literally vital for the arid and semi‐arid regions of the planet. Yet good water management requires multidisciplinary expertise, since one must consider climatic, hydrological, economical and social aspects to make balanced decisions on water usage. We here present SegHidro, a Grid portal designed to foster scientific, technical and operational collaboration to improve water resources management. The portal targets researchers and decision makers, enabling them to execute and couple their computational models in a workflow. The portal provides a framework which allows seamless integration of the models, meaning that each phase of the flow may be executed by a different expert and that the resulting data are shared among other portal users. Due to the nature of these applications and the need to execute many prospective scenarios, their execution requires high computing power. However, we go beyond providing high‐performance computational Grid capabilities. We also enable people to complement each other's expertise in understanding the trade‐offs in the water allocation decisions. The SegHidro portal is about sharing: human expertise, data and computing power. Copyright © 2007 John Wiley & Sons, Ltd.

Grids have the potential to revolutionize computing by providing ubiquitous, on demand access to computational services and resources. However, grid systems are extremely large, complex and prone to failures. A survey we have conducted reveals that fault diagnosis is still a major problem for grid users. When a failure appears at the user screen, it becomes very difficult for the user to identify whether the problem is in his application, somewhere in the grid middleware, or even lower in the fabric that comprises the grid. To overcome this problem, we argue that current grid platforms must be augmented with a collaborative diagnosis mechanism. We propose for such mechanism to use automated tests to identify the root cause of a failure and propose the appropriate fix. We also present a Java-based implementation of the proposed mechanism, which provides a simple and flexible framework that eases the development and maintenance of the automated tests.

Software testing is a fundamental part of system development. As software grows, its test suite becomes larger and its execution time may become a problem to software developers. This is especially the case for agile methodologies, which preach a short develop/test cycle. Moreover, due to the increasing complexity of systems, there is the need to test software in a variety of environments. In this paper, we introduce GridUnit, an extension of the widely adopted JUnit testing framework, able to automatically distribute the execution of software tests on a computational grid with minimum user intervention. Experiments conducted with this solution have showed a speed-up of almost 70x, reducing the duration of the test phase of a synthetic application from 24 hours to less than 30 minutes. The solution does not require any source-code modification, hides the grid complexity from the user and provides a cost-effectiveness improvement to the software testing experience.

As Web services and service oriented architectures are adopted, it is increasingly important to have standard and interoperable means to deploy and configure Web services. Within the Global Grid Forum, HP, NEC, and Softricity have been developing a standard for configuration description, deployment, and lifecycle management (CDDLM). In order to prove its feasibility, reference implementations are being developed. This paper describes an independent reference implementation of CDDLM and the experience in using Web Services for deployment in a standardized manner. Our main contributions are: the lessons learned in implementing this WS-based standard and an architecture for implementing CDDLM

In this paper we study reciprocation-based mechanisms to encourage donation in peer-to-peer grids in which multiple services, such as processing power and data transfers, are shared explicitly. We have modeled such a system and established how peers should assess whether it is profitable to exchange services with another peer, an issue that is not present in the single service case. Unfortunately, this assessment relies on information provided by untrustworthy peers. As an alternative, we have extended, to the case of multiple services, a reciprocation-based mechanism which uses only reliable information gathered locally. We have assessed this mechanism by simulating scenarios in which services are exchanged that are combinations of two different basic services. In the explored scenarios the mechanism performs very well, and can marginalize free riders even when the cost to peers of donating a service is nearly as large as the utility gained by receiving it

This paper presents and evaluates a governmental application that has been ported to run in a Grid. The Gerpav application is used in the city of Porto Alegre, located in the south of Brazil, to maintain and plan the investments in the city road system, and Grid technology brought substantial performance gains to its distributed version called GerpavGrid. We describe several optimizations strategies used in GerpavGrid to move the bottleneck of the sequential application from database to memory to facilitate distribution of tasks in the Grid. Results of its execution in a real Grid are also presented, and show that a Grid can be also an interesting execution platform for non-classical HPC applications that are database intensive.

This paper describes our experience with SegHidro, a project that empowers hydro-meteorological researchers by (i) enabling collaborative work via the coupling of computer models, and (ii) providing access to massive grid-based computer resources. SegHidro researchers are geographically distributed and have different and sometimes complementary backgrounds. They have in common their interest over the Brazilian Northeast, a semi-arid region, where irregular rainfall distribution causes many problems to the population. SegHidro was created out of the need to help decision makers to better manage water resources. It relies on grid computing as the "glue" that sticks together the community resources: data, computing power and human expertise. Our experience has shown that simplicity is key for the adoption of a solution and its success. Thus SegHidro is built over a simple infrastructure in both computation and data

Clusters of workstations are one of the most suitable resources to assist e-scientists in the execution of large-scale experiments that demand processing power. The utilization rate of these machines is usually far from 100%, and hence this should motivate administrators to share their clusters to grid communities. However, exploiting these resources in computational grids is challenging and brings several problems. This paper presents a transparent resource allocation strategy to harness idle cluster resources aimed at executing grid applications. This novel approach does not make use of a formal allocation request to cluster resource managers. Moreover, it does not interfere with local cluster users, being non-intrusive, and hence motivating cluster administrators to publish their resources to grid communities. We present experimental results regarding the effects of the proposed strategy on the attendance time of both cluster and grid requests and we also analyze its effectiveness in clusters with different utilization rates.

Grid computing is turning from promise into reality. Naturally, however, this is not happening for all applications at once. Bag-of-tasks (BoT) applications (those parallel applications whose tasks are independent) are, due to their simplicity, the first class of applications to be massively executed on grids (e.g. SETI@home). BoT applications are especially amendable for grid execution because they can run on intermittent resources (i.e. resources with no guarantees on availability or reliability). In this scenario, good performance and reliable results are provided by eager schedulers, which rely on replication to overcome unfortunate task-to-processor assignments. A little surprisingly, however, eager schedulers are not prepared to use space-shared resources (e.g. parallel supercomputers). This happens because using a space-shared resource involves submitting a detailed request to the resource scheduler, specifying the number of processors needed and the amount of time these processors are to be allocated - information that eager schedulers are not prepared to provide. This is unfortunate because space-shared computers are the most powerful computing resources available today and thus could greatly improve the execution time of BoT applications. This work proposes an automatic way to craft such requests in order to convert space-shared resources into intermittent ones, therefore rendering them naturally usable by eager schedulers. Such conversion is based on adaptive heuristics and allows eager schedulers use such powerful computing resources without modifications.

We investigate the circumstances under which it is possible to discourage free-riding in a peer-to-peer system for resource-sharing by prioritizing resource allocation to peers with higher reputation. We use a model to predict conditions necessary for any reputation scheme to succeed in discouraging free-riding by this method. We show with simulations that for representative cases, a very simple autonomous reputation scheme works nearly as well at discouraging free-riding as an ideal reputation scheme. Finally, we investigate the expected dynamic behavior of the system.

In this paper we discuss the difficulties involved in the scheduling of applications on computational grids. We highlight two main sources of difficulties: 1) the size of the grid rules out the possibility of using a centralized scheduler; 2) since resources are managed by different parties, the scheduler must consider several different policies. Thus, we argue that scheduling applications on a grid require the orchestration of several schedulers, with possibly conflicting goals. We discuss how we have addressed this issue in the context of PAUA, a grid for Bag-of-Tasks applications (i.e. parallel applications whose tasks are independent) that we are currently deploying throughout Brazil.

In this paper, we analyze the performance of consensus protocols based on the rotating coordinator paradigm. We consider a simulated production environment, on which processing and communication resources available for the different processes running the protocols are not necessarily the same. Firstly, we show that, in some scenarios, the performance of the consensus protocol is enhanced when there is an increase in the number and duration of the wrong suspicions periods of the failure detection service used. Since it is well known that wrong suspicions may also decrease the performance of the consensus protocol, a new dilemma is posed to the designers of such protocols. We then propose a new approach to address performance issues in the design of crash-detection based distributed protocols for asynchronous systems. We argue that they must be designed to adapt themselves to the variations on the availability of resources. The concept of slowness oracles is proposed to achieve this goal. Finally, we present a slowness oracle that can be used to transform a non-adaptive consensus protocol into an adaptive one. Simulations show that the adaptive protocol outperforms its conventional non-adaptive counterpart in a number of scenarios, having an equivalent performance in the other scenarios.

We here discuss how to run Bag-of-Tasks applications on computational grids. Bag-of-Tasks applications (those parallel applications whose tasks are independent) are both relevant and amendable for execution on grids. However, few users currently execute their Bag-of-Tasks applications on grids. We investigate the reason for this state of affairs and introduce MyGrid, a system designed to overcome the identified difficulties. MyGrid provides a simple, complete and secure way for a user to run Bag-of-Tasks applications on all resources she has access to. Besides putting together a complete solution useful for real users, MyGrid embeds two important research contributions to grid computing. First, we introduce some simple working environment abstractions that hide machine configuration heterogeneity from the user. Second, we introduce work queue with replication (WQR), a scheduling heuristics that attains good performance without relying on information about the grid or the application, although consuming a few more cycles. Note that not depending on information makes WQR much easier to deploy in practice.

Ensembles of distributed, heterogeneous resources, also known as computational grids, have emerged as critical platforms for high-performance and resource-intensive applications. Such platforms provide the potential for applications to aggregate enormous bandwidth, computational power, memory, secondary storage, and other resources during a single execution. However, achieving this performance potential in dynamic, heterogeneous environments is challenging. Recent experience with distributed applications indicates that adaptivity is fundamental to achieving application performance in dynamic grid environments. The AppLeS (Application Level Scheduling) project provides a methodology, application software, and software environments for adaptively scheduling and deploying applications in heterogeneous, multiuser grid environments. We discuss the AppLeS project and outline our findings.

In most parallel supercomputers, submitting a job for execution involves specifying how many processors are to be allocated to the job. When the job is moldable (i.e., there is a choice on how many processors the job uses), an application scheduler called SA can significantly improve job performance by automatically selecting how many processors to use. Since most jobs are moldable, this result has great impact to the current state of practice in supercomputer scheduling. However, the widespread use of SA can change the nature of workload processed by supercomputers. When many SAs are scheduling jobs on one supercomputer, the decision made by one SA affects the state of the system, therefore impacting other instances of SA. In this case, the global behavior of the system comes from the aggregate behavior caused by all SAs. In particular, it is reasonable to expect the competition for resources to become tougher with multiple SAs, and this tough competition to decrease the performance improvement attained by each SA individually. This paper investigates this very issue. We found that the increased competition indeed makes it harder for each individual instance of SA to improve job performance. Nevertheless, there are two other aggregate behaviors that override increased competition when the system load is moderate to heavy. First, as load goes up, SA chooses smaller requests, which increases efficiency, which effectively decreases the offered load, which mitigates long wait times. Second, better job packing and fewer jobs in the system make it easier for incoming jobs to fit in the supercomputer schedule, thus reducing wait times further. As a result, in moderate to heavy load conditions, a single instance of SA benefits from the fact that other jobs are also using SA.

The performance of supercomputer schedulers is influenced by the workloads that serve as their input. Realistic workloads are therefore critical to evaluate how supercomputer schedulers perform in practice. There has been much written in the literature about rigid parallel jobs, i.e. jobs that require partitions of a fixed size to run. However the majority of the parallel jobs in production today are moldable, i.e. jobs that can execute on a variety of partition sizes. In this paper we describe a workload model for moldable jobs, which is based on a user survey and good analytical models. Our model can serve as the basis for the development of performance-efficient strategies for selection of the job partition size, as well as the basis for enhancing supercomputer schedulers to directly accept moldable request.