The Common Vulnerability Scoring System (CVSS) is the industry standard for describing the characteristics of a software vulnerability and measuring its severity. However, during the first days after a vulnerability disclosure, the initial human readable description of the vulnerability is not available as a machine readable CVSS vector yet. This situation creates a period of time when only expensive manual analysis can be used to react to new vulnerabilities because no data is available for cheaper automated analysis yet. We present a new technique based on linear regression to automatically predict the CVSS vector of newly disclosed vulnerabilities using only their human readable descriptions, with a strong emphasis on decision explicability. Our experimental results suggest real world applicability.

Common Vulnerabilities and Exposures (CVE) databases such as Mitre’s CVE List and NIST’s NVD database identify every disclosed vulnerability affecting any public software. However, during the early hours of a vulnerability disclosure, the metadata associated with these vulnerabilities is either missing, wrong, or at best sparse. This creates a challenge for robust automated analysis of new vulnerabilities. We present a new technique based on TF-IDF to assess the software products most probably affected by newly disclosed vulnerabilities, formulated as an ordered list of relevant keywords. For doing so we rely only on the human readable description of a new vulnerability without any need for its metadata. Our evaluation results suggest real world applicability of our technique.

Summary The cloud has emerged as an attractive platform for resource‐intensive scientific applications, such as epidemic simulators. However, building and executing such applications in the cloud presents multiple challenges, including exploiting elasticity, handling failures, and simplifying multi‐cloud deployment. To address these challenges, this paper proposes a novel service‐based framework called DiFFuSE that enables simulation applications with a bag‐of‐tasks structure to fully exploit cloud platforms. This paper describes how the framework is applied to restructure two legacy applications, simulating the spread of bovine viral diarrhea virus and Mycobacterium avium subspecies paratuberculosis, into elastic cloud‐native applications. Experimental results show that the framework enhances application performance and allows exploring different cost‐performance trade‐offs while supporting automatic failure handling and elastic resource acquisition from multiple clouds.

With the advent of the Internet of Things (IoT), data stream processing have gained increased attention due to the ever-increasing need to process heterogeneous and voluminous data streams. This work addresses the problem of selecting a correct stream processing framework for a given application to be executed within a specific physical infrastructure. For this purpose, we focus on a thorough comparative analysis of three data stream processing platforms - Apache Flink, Apache Storm, and Twitter Heron (the enhanced version of Apache Storm), that are chosen based on their potential to process both streams and batches in real-time. The goal of the work is to enlighten the cloud-clients and the cloud-providers with the knowledge of the choice of the resource-efficient and requirement-adaptive streaming platform for a given application so that they can plan during allocation or assignment of Virtual Machines for application execution. For the comparative performance analysis of the chosen platforms, we have experimented using 8-node clusters on Grid5000 experimentation testbed and have selected a wide variety of applications ranging from a conventional benchmark to sensor-based IoT application and statistical batch processing application. In addition to the various performance metrics related to the elasticity and resource usage of the platforms, this work presents a comparative study of the “green-ness” of the streaming platforms by analyzing their power consumption - one of the first attempts of its kind. The obtained results are thoroughly analyzed to illustrate the functional behavior of these platforms under different computing scenarios.

IaaS clouds allow customers (called tenants) to deploy their IT as virtualized infrastructures. However IaaS clouds features, such as multi-tenancy and elasticity, generate new security vulnerabilities for which the security monitoring must be partly run by the cloud provider to give visibility at the virtualization infrastructure level. Unfortunately the same IaaS clouds features make the virtualized infrastructures frequently reconfigurable and thus affect the ability of a provider-run security monitoring system to detect attacks.

Infrastructure as a Service (IaaS) clouds are very dynamic with at runtime frequent changes at different levels of the virtual infrastructure. For cloud tenants, this affects the ability of a security monitoring framework to successfully detect attacks. In this paper, we propose SAIDS, a self-adaptable intrusion detection system for IaaS clouds that is able to adapt its components based on dynamic events that occur in a cloud infrastructure. We implemented and experimentally evaluated SAIDS, and show that it is a scalable solution that successfully detects attacks even during the adaptation process while imposing negligible overhead to cloud operations and tenant applications.

Reducing the energy consumed by datacenters becomes of major importance due to the current climate changes and the increasing success of cloud computing. Studies to optimize the energy consumption of cloud systems exist but do not take into account the end-user. The goal of this paper is to understand and quantify the link between the PaaS parameters a user can configure and the application energy consumption. In this work we summarize the parameters available at the PaaS layer and measure their influence on the energy consumed by RUBiS, a web application benchmark. Different database technologies and programming languages are compared as well as their software versions. Experimentation results show that by themselves the existing PaaS parameters can variate the application energy consumption. Although it shows that different programming languages do not consume the same, we discovered that a wider energy difference exists between database technologies which means higher energy savings is possible when the less consuming technology is used. Thus, users could optimize the energy impact of their applications by carefully configuring on-hand PaaS parameters.

In an IaaS cloud the physical infrastructure is controlled by service providers, including its security monitoring aspect. Clients hosting their information system are incited to trust the provider's claim (e.g. infrastructure availability) thanks to the assurance given by Service Level Agreements (SLAs). We aim at extending SLAs to include security monitoring terms. In this paper we describe the challenges to reach this goal, we propose a three-step incremental strategy and we apply the first step of this strategy on the case of network IDS (NIDS) monitoring probes. In this case study we select a relevant metric to describe the performance of an NIDS, that is the metric can figure in an SLA and can be measured to verify that the SLA is respected. In particular we propose an in situ verification method of such a metric on a production NIDS and evaluate experimentally and analytically the proposed method.

Applications are increasingly being deployed on resources delivered by Infrastructure-as-a-Service (IaaS) cloud providers. A major challenge for application owners is continually managing the application deployment in order to satisfy the performance requirements of application users while reducing the charges paid to IaaS providers. This paper proposes an approach for adaptive application deployment that explicitly considers adaptation costs and benefits in making deployment decisions. The approach builds on the PaaSage open-source platform, thus enabling automatic deployment and execution over multiple clouds. The paper describes experiments in a real cloud testbed that demonstrate that the approach enables multi-cloud adaptation while increasing the total value of the application for its owner.

Reducing the massive amount of energy consumed by cloud datacenters becomes of major importance. In the usual approach where resources are consolidated into fewer servers in order to power down the others, it still remains periods of time when servers are not fully utilized. Consequently, it exists unused resources that are not exploited although they could be used to execute applications compatible with the variable availability of these resources. In this work, we propose a cloud system where the Platform-as-a-Service (PaaS) and Infrastructure-as-a-Service (IaaS) layers interact to find execution trade-offs that exploit the unused resources at IaaS level. PaaS users are involved in the energy optimization by proposing to delay their executions and adapt resource sizes in order to fit with the available unused resources. Our evaluation by simulation is based on real data and expresses a realistic large scale cloud scenario. Results show that according to the proportion of energy-aware users, this system is able to reduce the amount of servers by using resources that would have been wasted otherwise. Therefore, our solution allows datacenters to consume less energy than with usual resource managers where all applications start their execution at submission time with their initial resource size.

Performing large-scale epidemic simulations is time-consuming and requires massive amounts of computing resources. An attractive option for providing such resources is cloud computing. However, fully exploiting cloud resources requires converting existing simulation applications into elastic, cloud-enabled applications. To this end, this paper proposes a novel service-based framework for building and executing epidemic simulations in the cloud and describes how the framework was applied to build a cloud-enabled simulation of the spread of the bovine viral diarrhea virus. The paper provides experimental evidence that the framework increases performance and allows exploring cost-performance trade-offs while automatically handling failures and supporting elastic allocation of resources from multiple clouds.

For management and security purposes, cloud providers should know the connectivity graph between virtual machines. Since traditional methods used in physical networks produce incomplete results and are hardly usable in the Cloud, we propose to use information provided by a Cloud Management Software and an SDN controller, to compute the connectivity graph in those environments. Our approach shows an exact, complete and up-to-date connectivity graphs computation on a representative infrastructure, in reasonable time.

Cloud providers have an incomplete view of their hosted virtual infrastructures managed by a Cloud Management System (CMS) and a Software Defined Network (SDN) controller. For various security reasons (e.g. isolation verification, modeling attack paths in the network), it is necessary to know which virtual machines can interact via network protocols. This requires building a connectivity graph between the virtual machines, that we can extract with the knowledge of the overall topology and the deployed network security policy. Existing methodologies for building such models for physical networks produce incomplete results. Moreover, they are not suitable for cloud infrastructures due to either their intrusiveness or lack of connectivity discovery. We propose a method to compute the connectivity graph, relying on information provided by both the CMS and the SDN controller. Connectivity can first be extracted from knowledge databases, then dynamically updated on the occurrence of cloud-related events. This approach shows an exact, complete and up-to-date connectivity graphs computation on a representative infrastructure, in reasonable time.

Cloud computing has become an attractive and easy-to-use solution for users who want to externalize the run of their applications. However, data centers hosting cloud systems consume enormous amounts of energy. Reducing this consumption becomes an urgent challenge with the rapid growth of cloud utilization. In this paper, we explore a way for energy-aware HPC cloud users to reduce their footprint on cloud infrastructures by reducing the size of the virtual resources they are asking for. We study the influence of green users on the system energy consumption and compare it with the consumption of more aggressive users in terms of resource utilization. We found that larger resources are more energy demanding even if they are faster in executing the applications. But, reducing too much the resources' size is also not beneficial for the energy consumption. A tradeoff lies in between these two options.

Summary The socioeconomic impact of cloud solutions and particularly PaaS (platform as a service) solutions is becoming increasingly critical. The main objective of PaaS providers is to generate the maximum profit from the services they provide. This requires them to face a number of challenges such as efficiently managing the underlying resources and satisfying the SLAs of the hosted applications. In this paper, we consider a cloud‐bursting PaaS environment where the PaaS provider owns some private resources and is able to rent public cloud resources on demand. In that context, we propose a profit‐efficient solution for managing such a system under SLA constraints. We define a profit optimization policy that evaluates the cost of hosting each application using public and private resources and chooses the option that generates the highest profit. During peak periods, the optimization policy considers two more options. The first option is to obtain some resources from running applications, taking into account the payment of penalties if their promised quality of service is affected. The second option is to wait until private resources become available, taking into account the payment of penalties if the quality of service promised to the new application is affected. Furthermore, we designed and implemented an open cloud‐bursting PaaS system, called Meryn, which integrates the proposed optimization policy and provides support for batch and MapReduce applications. The results of our evaluation show the effectiveness of our approach in optimizing the provider profit. Indeed, compared to a basic approach, our approach provides up to 11.59% and 9.02% more provider profit in, respectively, simulations and experiments.

Application-level firewalls filter traffic based on a white list of processes that are allowed to access the network. Although they have a complete overview of the system in which they are executed, they can be easily bypassed by knowledgable attackers. In this paper we present AL-SAFE, a cloud-tailored application-level self-adaptable firewall which combines the high degree of visibility of an application-level firewall with the isolation of a traditional standalone firewall. AL-SAFE is able to filter traffic at two distinct points in the virtual infrastructure and adapt the enforced rulesets based on changes in the virtual infrastructure topology and the list of services running inside the virtual machines. Our performance analysis shows that AL-SAFE imposes a tolerable delay to legitimate network connections while it is able to filter out all unauthorised packets.

Service providers give assurance on some aspects of the service but, as of today, security monitoring is not one of them. In our work, we aim to allow providers to provide customers with guarantees on security monitoring of their outsourced information system.

A non-invasive, cloud-agnostic approach is demonstrated for extending existing cloud platforms to include checkpoint-restart capability. Most cloud platforms currently rely on each application to provide its own fault tolerance. A uniform mechanism within the cloud itself serves two purposes: (a) direct support for long-running jobs, which would otherwise require a custom fault-tolerant mechanism for each application, and (b) the administrative capability to manage an over-subscribed cloud by temporarily swapping out jobs when higher priority jobs arrive. An advantage of this uniform approach is that it also supports parallel and distributed computations, over both TCP and InfiniBand, thus allowing traditional HPC applications to take advantage of an existing cloud infrastructure. Additionally, an integrated health-monitoring mechanism detects when long-running jobs either fail or incur exceptionally low performance, perhaps due to resource starvation, and proactively suspends the job. The cloud-agnostic feature is demonstrated by applying the implementation to two very different cloud platforms: Snooze and Open Stack. The use of a cloud-agnostic architecture also enables, for the first time, migration of applications from one cloud platform to another.

Traditional intrusion detection systems are not adaptive enough to cope with the dynamic characteristics of cloud-hosted virtual infrastructures. This makes them unable to address new cloud-oriented security issues. In this paper we introduce SAIDS, a self-adaptable intrusion detection system tailored for cloud environments. SAIDS is designed to re-configure its components based on environmental changes. A prototype of SAIDS is described.

PaaS (Platform as a Service) is an increasingly popular cloud model, providing a complete development and hosting environment for cloud applications. As the use of PaaS becomes pervasive, defining and maintaining SLAs (Service Level Agreements) between PaaS customers and providers becomes essential. Useful SLAs should provide guarantees on application quality properties (e.g., response time) rather than on resource availability (e.g., number of virtual machines). Current PaaS offerings either provide no support for providing such guarantees or provide support targeting a restricted set of application types. In this paper, we present an SLA-driven PaaS architecture, called Meryn, which supports cloud bursting and is designed to be easily extensible to host new application types. We propose a decentralized optimization policy aiming at maximizing the PaaS provider profit and taking into account the payment of penalties incurred when quality guarantees are unsatisfied. In addition, we apply the proposed optimization policy to batch and MapReduce applications. We implemented and evaluated our policy through a series of simulations on the Grid5000 test bed. The results show that our approach provides up to 14.77% more profit for the provider and uses up to 80.99% less public clouds resources compared with a basic approach.

The exponential growth of scientific and business data has resulted in the evolution of the cloud computing and the MapReduce parallel programming model. Cloud computing emphasizes increased utilization and power savings through consolidation while MapReduce enables large scale data analysis. The Hadoop framework has recently evolved to the standard framework implementing the MapReduce model. In this paper, we evaluate Hadoop performance in both the traditional model of collocated data and compute services as well as consider the impact of separating out the services. The separation of data and compute services provides more flexibility in environments where data locality might not have a considerable impact such as virtualized environments and clusters with advanced networks. In this paper, we also conduct an energy efficiency evaluation of Hadoop on physical and virtual clusters in different configurations. Our extensive evaluation shows that: (1) performance on physical clusters is significantly better than on virtual clusters; (2) performance degradation due to separation of the services depends on the data to compute ratio; (3) application completion progress correlates with the power consumption and power consumption is heavily application specific.

The MapReduce programming model offers a simple and efficient way of performing distributed computation over large data sets. To enable the usage of MapReduce in the cloud, Amazon Web Services offers Elastic MapReduce (EMR), a web service enabling users to easily run MapReduce jobs by leveraging Amazon resources (i.e. compute and storage). EMR takes care of tasks such as resource provisioning, performance tuning, and fault tolerance thus allowing the users to concentrate on the problem to be solved. However, EMR is restricted to Amazon's resources and is provided at an additional cost. In this paper, we present the design, implementation, and evaluation of Resilin, a novel EMR API-compatible system to perform distributed MapReduce computations. Resilin goes one step beyond Amazon's proprietary EMR solution and allows users (e.g. companies, scientists) to leverage resources from one or multiple public and/or private clouds. This gives Resilin users the opportunity to perform MapReduce computations over a large number of potentially geographically distributed resources. An extensive experimental evaluation conducted on multiple clusters of the Grid'5000 experimentation test bed shows that Resilin enables the use of geographically distributed resources with only limited impact on MapReduce jobs execution time.

Snooze is an open-source scalable, autonomic, and energy-efficient virtual machine (VM) management framework for private clouds. It allows users to build compute infrastructures from virtualized resources. Particularly, once installed and configured, it allows its users to submit and control the life-cycle of a large number of VMs. For scalability, the system relies on a self-organizing hierarchical architecture. Moreover, it implements self-healing mechanisms in case of failure to enable high availability. It also performs energy-efficient distributed VM management through consolidation and power management techniques. This poster focuses on the experimental validation of two main properties of Snooze: scalability and fault-tolerance.

Cloud computing brings new service models that offer users more flexibility and better resource control promising to improve the user's satisfaction in terms of Quality of Service. However, current cloud solutions, in particular above the Infrastructure-as-a-Service layer, don't provide automated solutions that address the Quality of Service users may want for their applications. In this paper we present Merkat, a platform that is capable to dynamically share cloud resources between the different application types using a market-based resource allocation mechanism. Merkat provides fair and maximum resource utilization by allocating fine-grained resource amounts to users proportionally to their communicated value. In the same time, autonomous controllers provide Service Level Objective support by applying elasticity rules to scale the application demand according to user performance objectives. Our experiments show that Merkat is able to efficiently react to changes in application load and user requirements, leading to better user satisfaction.

SUMMARY Live virtual machine migration is a powerful feature of virtualization technologies. It enables efficient load balancing, reduces energy consumption through dynamic consolidation, and makes infrastructure maintenance transparent to users. Although live migration is available across wide area networks with state of the art systems, it remains expensive to use because of the large amounts of data to transfer, especially when migrating virtual clusters rather than single virtual machine instances. As evidenced by previous research, virtual machines running identical or similar operating systems have significant portions of their memory and storage containing identical data. We propose Shrinker, a live virtual machine migration system leveraging this common data to improve live virtual cluster migration between data centers interconnected by wide area networks. Shrinker detects memory pages and disk blocks duplicated in a virtual cluster to avoid sending the same content multiple times over wide‐area network links. Virtual machine data is retrieved in the destination site with distributed content‐based addressing. We implemented a prototype of Shrinker in the KVM (Kernel‐based Virtual Machine) hypervisor and present a performance evaluation in a distributed environment. Experiments show that it reduces both total data transferred and total migration time. Copyright © 2012 John Wiley & Sons, Ltd.

Energy efficiency has now become one of the major design constraints for current and future cloud data center operators. One way to conserve energy is to transition idle servers into a lower power-state (e.g. suspend). Therefore, virtual machine (VM) placement and dynamic VM scheduling algorithms are proposed to facilitate the creation of idle times. However, these algorithms are rarely integrated in a holistic approach and experimentally evaluated in a realistic environment. In this paper we present the energy management algorithms and mechanisms of a novel holistic energy-aware VM management framework for private clouds called Snooze. We conduct an extensive evaluation of the energy and performance implications of our system on 34 power-metered machines of the Grid'5000 experimentation testbed under dynamic web workloads. The results show that the energy saving mechanisms allow Snooze to dynamically scale data center energy consumption proportionally to the load, thus achieving substantial energy savings with only limited impact on application performance.

With the advent of cloud computing and the need to satisfy growing customers resource demands, cloud providers now operate increasing amounts of large data centers. In order to ease the creation of private clouds, several open-source Infrastructure-as-a-Service (IaaS) cloud management frameworks (e.g. Open Nebula, Nimbus, Eucalyptus, Open Stack) have been proposed. However, all these systems are either highly centralized or have limited fault tolerance support. Consequently, they all share common drawbacks: scalability is limited by a single master node and Single Point of Failure (SPOF). In this paper, we present the design, implementation and evaluation of a novel scalable and autonomic (i.e. self-organizing and healing) virtual machine (VM) management framework called Snooze. For scalability the system utilizes a self-organizing hierarchical architecture and performs distributed VM management. Moreover, fault tolerance is provided at all levels of the hierarchy, thus allowing the system to self-heal in case of failures. Our evaluation conducted on 144 physical machines of the Grid'5000 experimental test bed shows that the fault tolerance features of the framework do not impact application performance. Moreover, negligible cost is involved in performing distributed VM management and the system remains highly scalable with increasing amounts of resources.

One way to conserve energy in cloud data centers is to transition idle servers into a power saving state during periods of low utilization. Dynamic virtual machine (VM) consolidation (VMC) algorithms are proposed to create idle times by periodically repacking VMs on the least number of physical machines (PMs). Existing works mostly apply VMC on top of centralized, hierarchical, or ring-based system topologies which result in poor scalability and/or packing efficiency with increasing number of PMs and VMs. In this paper, we propose a novel fully decentralized dynamic VMC schema based on an unstructured peer-to-peer (P2P) network of PMs. The proposed schema is validated using three well known VMC algorithms: First-Fit Decreasing (FFD), Sercon, V-MAN, and a novel migration-cost aware ACO-based algorithm. Extensive experiments performed on the Grid'5000 testbed show that once integrated in our fully decentralized VMC schema, traditional VMC algorithms achieve a global packing efficiency very close to a centralized system. Moreover, the system remains scalable with increasing number of PMs and VMs. Finally, the migration-cost aware ACO-based algorithm outperforms FFD and Sercon in the number of released PMs and requires less migrations than FFD and V-MAN.

SUMMARY Message logging is an attractive solution to provide fault tolerance for message‐passing applications because it is more scalable than coordinated checkpointing. Sender‐based message logging is a well‐known optimization that allows the saving of message payload in the sender memory. Thus, only message reception events have to be logged reliably by using an event logger. This paper proposes solutions to further improve message logging protocol scalability. In existing works on message logging, the event logger has always been considered as a centralized process. We propose a distributed event logger that takes advantage of multi‐core processors that are to be executed in parallel with application processes, leveraging the volatile memory of the nodes to save events reliably. We also propose the combination of our distributed event logger and O2P, an active optimistic message logging protocol using a gossip‐based protocol to disseminate information on new stable events. Our distributed event logger and O2P are implemented in the Open MPI library. Our results show the following: (i) distributed event logging improves message logging protocol scalability and (ii) using O2P with a distributed event logger provides an efficient and scalable fault‐tolerant solution for message‐passing applications. Copyright © 2011 John Wiley & Sons, Ltd.

Virtualization technologies have recently gained a lot of interest in Grid computing as they allow flexible resource management. However, the most common way to exploit grids relies on dedicated services like resource management systems (RMSs) to get resources at a particular time. To improve resource usage, most of these systems provide a best-effort mode where lowest priority jobs can be executed when resources are idle. This particular mode does not provide any guarantee of service and jobs may be killed at any time by the RMS when the nodes they use are subject to higher priority reservations. This behaviour potentially leads to a huge waste of computation time or at least requires users to deal with checkpoints of their jobs. In this paper we present Saline, a generic and non-intrusive framework to manage best-effort jobs at grid level through virtual machines (VMs) usage. We discuss the main challenges concerning the design of such a grid system, focusing on VM snapshot management and network configuration. Results of experiments show our proposal ensures an efficient execution of best-effort jobs through the whole grid.

Active replication on top of a structured peer-to-peer overlay is an attractive solution for transparently providing high availability to distributed applications. However, self-healing is necessary to ensure the availability of the replicated application despite node arrivals, failures or departures in the overlay. Self-healing means to automatically reconfigure the replica groups when changes in the overlay occur. In the case of active replication, reconfigurations must be done carefully, to keep the replicas consistency. Moreover, as every reconfiguration could imply a state transfer between replicas, their number should be limited. In this paper we propose a self-healing solution that limits the number of group reconfigurations and ensures the availability of the replicated application in a dynamic environment. To evaluate the performance of our solution, we implemented it in a framework, called Semias, in the context of Vigne grid middleware. Experiments run on Grid'5000 and Planet Lab show the performance of the framework and the efficiency of our self-healing mechanisms in a dynamic environment.

Computer clusters are today the reference architecture for high-performance computing. The large number of nodes in these systems induces a high failure rate. This makes fault tolerance mechanisms, e.g. process checkpoint/restart, a required technology to effectively exploit clusters. Most of the process checkpoint/restart implementations only handle volatile states and do not take into account persistent states of applications, which can lead to incoherent application restarts. In this paper, we introduce an efficient persistent state checkpoint/restoration approach that can be interconnected with a large number of file systems. To avoid the performance issues of a stable support relying on synchronous replication mechanisms, we present a failure resilience scheme optimized for such persistent state checkpointing techniques in a distributed environment. First evaluations of our implementation in the kDFS distributed file system show the negligible performance impact of our proposal.

With the growing scale of high performance computing platforms, fault tolerance has become a major issue. Among the various approaches for providing fault tolerance to MPI applications, message logging has been proved to tolerate higher failure rate. However, this advantage comes at the expense of a higher overhead on communications, due to latency intrusive logging of events to a stable storage. Previous work proposed and evaluated several protocols relaxing the synchronicity of event logging to moderate this overhead. Recently, the model of message logging has been refined to better match the reality of high performance network cards, where message receptions are decomposed in multiple interdependent events. According to this new model, deterministic and non-deterministic events are clearly discriminated, reducing the overhead induced by message logging. In this paper we compare, experimentally, a pessimistic and an optimistic message logging protocol, using this new model and implemented in the Open MPI library. Although pessimistic and optimistic message logging are, respectively, the most and less synchronous message logging paradigms, experiments show that most of the time their performance is comparable.

Fault tolerance is one of the key issues for large scale applications executed on high performance computing systems. In a cluster federation, clusters are gathered to provide huge computing power. To work efficiently on such systems, networks characteristics have to be taken into account: the latency between two nodes of different clusters is much higher than the latency between two nodes of the same cluster. In this paper, we present O2P-CF a message logging protocol well-suited to provide fault tolerance for message passing applications executed on cluster federations. O2P-CF is based on the combination of O2P, an extremely optimistic message logging protocol, with a pessimistic message logging protocol.

The EU-funded XtreemOS project implements a grid operating system transparently exploiting resources of virtual organizations through the standard POSIX interface. Grid checkpointing and restart requires to save and restore jobs executing in a distributed heterogeneous grid environment. The latter may spawn millions of grid nodes ( PCs, clusters, and mobile devices ) using different system-specific checkpointers saving and restoring application and kernel data structures for processes executing on a grid node. In this paper we shortly describe the XtreemOS grid checkpointing architecture and how we bridge the gap between the abstract grid and the system-specific checkpointers. Then we discuss how we keep track of processes and how different process grouping techniques are managed to ensure that all processes of a job and any further dependent ones can be checkpointed and restarted. Finally, we present how Linux control groups can be used to address resource isolation issues during the restart.

In computational grids, a virtual organization (VO) is a dynamic coupling of multiple Linux/Unix nodes for resource sharing under specific polices. Currently, VO support functionalities are generally implemented as grid middleware. However, the usability of grids is often impaired by the complexity of configuring and maintaining a new layer of security infrastructure as well as adapting to new interfaces of security enabled services. In this paper, we present an OS-level approach to provide native VO support functionalities, which is a part of XtreemOS project [18]. Our approach adopts pluggable frameworks existing in current OS as extension points to implement VO support, avoiding modification of kernel codes and easily turning traditional OSes into grid-aware ones. The performance evaluation of NAS parallel benchmarks (NPB) shows that our current implementation incurs trivial overhead on original systems.

The overall objective of the XtreemOS project is the design, implementation, evaluation and distribution of an open source grid operating system (named XtreemOS) with native support for virtual organizations (VO) and capable of running on a wide range of underlying platforms, from clusters to mobiles. The approach we propose is to investigate the design of a grid OS, XtreemOS, based on the Linux existing general purpose OS. A set of system services, extending those found in the traditional Linux, will provide users with all the grid capabilities associated with current grid middleware, but fully integrated into the OS. The underlying Linux will be extended as needed to support VOs spanning across many machines and to provide appropriate interfaces to the Grid OS services. Installed on each participating machine, the XtreemOS system will provide for the grid what an operating system offers for a single computer: abstraction from the hardware, and secure resource sharing between different users. It would thus considerably ease the work of users belonging to VOs by giving them (as far as possible) the illusion of using a traditional computer, and releasing them from dealing with the complex resource management issues of a grid environment. By integrating grid capabilities into the kernel, XtreemOS will also provide a robust, secure and easy to manage infrastructure for system administrators

Monitoring distributed applications executed on a computational Grid is challenging since they are executed on several heterogeneous nodes belonging to different administrative domains. An application monitoring service should provide users and administrators with useful and dependable data on the applications executed on the Grid. We present in this paper a grid application monitoring service designed to handle high availability and scalability issues. The service supplies information on application state, on failures and on re- source consumption. A set of transparent monitoring mechanisms are used according to grid node nature to effectively monitor the applications. Experiments on the Grid'5000 testbed show that the service provides dependable information with a minimal cost on Grid performances.

Using grid resources to execute scientific applications requiring a large amount of computing power is attractive but not easy from the user point of view. Vigne is a grid operating system designed to provide users with a simplified view of a grid. Vigne deals with the huge number of nodes in a large-scale grid and with the nodes' dynamic behavior by using peer-to-peer overlays as a keystone. In this paper, we show why it is highly desirable to use structured and unstructured peer-to-peer overlays for building the high-level services of Vigne grid operating system. To show the interest of our approach, we detail the features of two Vigne services built on top of peer-to-peer overlays. We also present experimental results obtained on the Grid'5000 testbed showing the scalability of the Vigne infrastructure based on overlays and its practical interest for the implementation of Vigne distributed services

This paper presents a comparative study of Kerrighed, openMosix and OpenSSI, three single system image (SSI) operating systems for clusters. This experimental study gives an overview of SSI features offered by these SSI and evaluates performance of such features.

We consider a transparent data sharing service for distributed applications in the grid. Our service may alleviate the burden of the user and the programmer to manage the distribution and the migration of data by transparently locating, caching, and managing the consistency of the data. To fit in a large scale and dynamic environment such as the grid, our data sharing service tolerates any number of reconfigurations (benign failures, arrivals and departures of sites) in the system's life time, and up to a fixed number of simultaneous reconfigurations of the system. This service relies on application backward error recovery and replication to ensure the liveness of the application. Reconfiguration has a high impact on data location mechanisms. Our service leverages an underlying structured overlay network to solve this issue. We have experimentally evaluated our service and present in this paper an analysis of the results.

Process management mechanisms (process duplication, migration and checkpoint/restart) are very useful for high performance and high availability in clustering systems. The single system image approach aims at providing a global process management service with mechanisms for process checkpoint, process migration and process duplication. In this context, a common mechanism for process virtualization is highly desirable but traditional operating systems do not provide such a mechanism. This paper presents a kernel service for process virtualization called ghost process, extending the Linux kernel. The ghost process mechanism has been implemented in the Kerrighed single system image based on Linux

Dependency tracking between communicating tasks is an important concept in backward error recovery for parallel applications. One can extend the traditional dependence tracking model for message passing systems to track dependencies between shared memory and task private states for shared memory applications. The objective of this paper is to analyze the issues generated by locks and barriers in parallel applications so that we can checkpoint tasks at any time (even when holding or waiting for locks and barriers). In particular we attempt to extend earlier dependency tracking mechanisms to locks and barriers. We address both coordinated and uncoordinated checkpointing schemes.

Cluster federations are attractive for executing applications like large scale code coupling. However faults may appear frequently in such architectures. Thus, checkpointing long-running applications is desirable to avoid to restart them from the beginning in the event of a node failure. To take into account the constraints of a cluster federation architecture, an hybrid checkpointing protocol is proposed. It uses global coordinated checkpointing inside clusters but only quasi-synchronous checkpointing techniques between clusters. The proposed protocol has been evaluated by simulation and fits well for applications that can be divided into modules with lots of communications within modules but few between them.

A working single system image distributed operating system is presented. Dubbed Kerrighed, it provides a unified approach and support to both the MPI and the shared memory programming models. The system is operational in a 16-processor cluster at the Institut de Recherche en Informatique et Systemes Aleatoires in Rennes, France. In this paper, the system is described with emphasis on its main contributing and distinguishing factors, namely its DSM based on memory containers, its flexible handling of scheduling and checkpointing strategies, and its efficient and unified communications layer. Because of the importance and popularity of data parallel applications in these systems, we present a brief discussion of the mapping of two well known and established data parallel algorithms. It is shown that ShearSort is remarkably well suited for the architecture/system pair as is the ever so popular and important two-dimensional fast Fourier transform. (2D FFT).

Summary form only given. Code coupling applications can be divided into communicating modules, that may be executed on different clusters in a cluster federation. As a cluster federation comprises of a large number of nodes, there is a high probability of a node failure. We propose a hierarchical checkpointing protocol that combines a synchronized checkpointing technique inside clusters and a communication-induced technique between clusters. This protocol fits to the characteristics of a cluster federation (large number of nodes, high latency and low bandwidth networking technologies between clusters). A preliminary performance evaluation performed using a discrete event simulator shows that the protocol is suitable for code coupling applications.

This paper describes issues in the design and implementation of checkpointing and recovery modules for the Kerrighed DSM cluster system. Our design is for a DSM supporting the sequential consistency model. The mechanisms are general enough to be used in a number of different checkpointing and recovery protocols. It is designed to support common optimizations for performance suggested in literature, while staying light-weight during fault free execution. We also present preliminary performance results of the current implementation.

Abstract Parallel single‐level store (PSLS) systems integrate a shared virtual memory and a parallel file system. They provide programmers with a global address space including both memory and file data. PSLS systems implemented in a cluster thus represent a natural support for long‐running parallel applications, combining both the natural shared memory programming model and a large and efficient file system. However, the need to tolerate failures in such a system increases with the size of applications. In this paper we present a highly‐available parallel single level store system (HA‐PSLS), which smoothly integrates a backward error recovery high‐availability mechanism into a PSLS system. Our system is able to tolerate multiple transient failures, a single permanent failure, and power cut failures affecting the whole cluster, without requiring any specialized hardware. For this purpose, HA‐PSLS relies on a high degree of integration (and reusability) of high‐availability and standard features. A prototype integrating our high‐availability support has been implemented and we show some performance results in the paper. Copyright © 2003 John Wiley & Sons, Ltd.

Clusters are attractive for executing sequential and parallel applications. However, there is a need to design a cluster distributed operating system to provide a Single System Image. A cluster operating system providing both a DSM system and load balancing is attractive for efficiently executing a workload of sequential applications and shared memory parallel applications. Gobelins is a distributed operating system dedicated to clusters that provides both a DSM system and a process migration mechanism to support load balancing. In this paper, we present the implementation of Gobelins process migration mechanism which exploits Gobelins kernel level DSM system. We show that Gobelins DSM allows to implement simply an efficient migration mechanism, that can be used to move processes or threads among cluster nodes. A prototype of Gobelins has been implemented. Some performance results are presented in this paper.

Clusters of SMPs are attractive for executing shared memory parallel applications, but reconciling high performance and ease of programming remains an open issue. A possible approach is to provide an efficient single system image (SSI) operating system giving the illusion of an SMP machine. The global management of all resources to provide a true SSI remains an open issue. We introduce the concept of container as a mechanism to unify global resource management at the lowest operating system level. Higher level operating system services such as virtual memory system and file cache can be easily implemented based on containers and transparently take benefit of the whole memory resource available in the cluster.

A parallel single level store system (PSLS) integrates a shared virtual memory and a parallel file system. Managing the data globally it provides programmers of scientific applications with the attractive shared memory programming model combined with a large and efficient file system in a cluster. We present a cheap and efficient two-level checkpointing approach enabling a PSLS to tolerate failures. The first level checkpointing algorithm is very efficient and saves data in memory but requires a large amount of memory space. When memories are saturated, an alternative algorithm, saving a checkpoint on disks is implemented. Performance results present the impact of different variants of the checkpointing algorithms.

A single-level store (SLS) integrating a shared virtual memory and a parallel file system with file mapping as its interface is attractive for the execution of high-performance applications in a cluster. However, the probability of a node reboot or failure is quite high. In this paper, we present the design of a highly available SLS system. Our approach combines checkpointing in memory and permanent checkpointing on disk in a cluster using all cluster memory and disk resources. Preliminary performance results show the applicability of the proposed approach for parallel applications with huge input/output requirements.

Distributed Shared Memory (DSM) architectures are attractive to execute high performance parallel applications. Made up of a large number of components, these architectures have however a high probability of failure. We propose a protocol to tolerate node failures in cache-based DSM architectures. The proposed solution is based on backward error recovery and consists of an extension to the existing coherence protocol to manage data used by processors for the computation and recovery data used for fault tolerance. This approach can be applied to both Cache Only Memory Architectures (COMA) and Shared Virtual Memory (SVM) systems. The implementation of the protocol in a COMA architecture has been evaluated by simulation. The protocol has also been implemented in an SVM system on a network of workstations. Both simulation results and measurements show that our solution is efficient and scalable.

In this paper we describe a distributed and dynamic architecture for network management of a heterogeneous distributed system which we have implemented in order to perform network management of our local distributed system. The architecture is highly generic in nature and can be extended to a large heterogeneous network. The architecture is distributed in nature as it provides for multiple managers acting simultaneously. It is rendered dynamic by the utilization of mobile agents for the implementation of network management functionalities. The network managers in our architecture utilize client-server technology and/or mobile agent technology as and when required depending on the functionality implemented and their location. In this context we have implemented a Mobile Network Manager (MNM) which is a location independent network manager. Mobile agents are furnished by MAGENTA (Mobile AGENT environment for distributed Applications) which we have implemented for this purpose.

In the light of the increasing throughput of local area networks, Networks Of Workstations (NOWs) which provide a Distributed Shared Memory (DSM) have become a convenient and cheaper alternative to parallel architectures in the framework of parallel scientific applications. However, the probability that a failure occurs in such a system made up of a large number of components must not be neglected, especially for long-running applications. This paper presents the design, implementation and performance evaluation of ICARE, a page-based recoverable DSM implemented on top of an ATM-based NOW running the CHORUS microkernel. ICARE relies on a Backward Error Recovery (BER) mechanism, and provides a way to combine both efficiency and high-availability. The fact that checkpoints are stored in volatile memory provides a low-cost fault-tolerance mechanism, as well as the opportunity to exploit the symbiotic relationship between the data replication implemented in DSM systems and that needed for fault-tolerance. Furthermore, ICARE efficiently implements transparent process rollback recovery. Performance evaluations show the efficiency of the ICARE prototype that implements the proposed algorithms. © 1998 John Wiley & Sons, Ltd.

Distributed Shared Virtual Memory (DSVM) systems provide a shared memory abstraction on distributed memory architectures. Such systems ease parallel application programming because the shared-memory programming model is often more natural than the message-passing paradigm. However, the probability of failure of a DSVM increases with the number of sites. Thus, fault tolerance mechanisms must be implemented in order to allow processes to continue their execution in the event of a failure. This paper gives an overview of recoverable DSVMs (RDSVMs) that provide a checkpointing mechanism to restart parallel computations in the event of a site failure.

Due to the increasing number of their components, Scalable Shared Memory Multiprocessors (SSMMs) have a very high probability of experiencing failures. Tolerating node failures therefore becomes very important for these architectures particularly if they must be used for long-running computations. In this paper, we show that the class of Cache Only Memory Architectures (COMA) are good candidates for building fault-tolerant SSMMs. A backward error recovery strategy can be implemented without significant hardware modification to previously proposed COMA by exploiting their standard replication mechanisms and extending the coherence protocol to transparently manage recovery data. Evaluation of the proposed fault-tolerant COMA is based on execution driven simulations using some of the Splash applications. We show that, for the simulated architecture, the performance degradation caused by fault-tolerance mechanisms varies from 5% in the best case to 35% in the worst case. The standard memory behavior is only slightly perturbed. Moreover, results also show that the proposed scheme preserves the architecture scalability and that the memory overhead remains low for parallel applications using mostly shared data.

This paper focuses on the problem of fault tolerance in shared memory multiprocessors, and describes an architecture designed for transparently tolerating processor failures. The Recoverable Shared Memory (RSM) is the novel component of this architecture, providing a hardware supported backward error recovery mechanism which minimizes the propagation of recovery when a processor fails. The RSM permits a shared memory multiprocessor to be constructed using standard caches and cache coherence protocols, and does not require any changes to be made to applications software. The performance of the recovery scheme supported by the RSM is evaluated and compared with other schemes that have been proposed for fault tolerant shared memory multiprocessors. The performance study has been conducted by simulation using address traces collected from real parallel applications.

COMAs (Cache Only Memory Architectures) are an interesting class of large scale shared memory multiprocessors. They extend the concepts of cache memories and shared virtual memory by using the local memories of the nodes as large caches for a single shared address space. Due to their large number of components, these architectures are particularly susceptible to hardware failures and so fault tolerance mechanisms have to be introduced to ensure a high availability. In this paper, we propose an implementation of backward error recovery in a COMA which minimizes performance degradation and requires little hardware modifications. This implementation uses the features of a COMA to implement a stable storage abstraction using the standard memories of the architecture. Recovery data are replicated and mixed with current data in node memories both of which are managed in a transparent way using an extended coherence protocol.

The authors address the problem of implementing the CSP (communicating sequential processes) rendezvous within a transactional framework. Instead of implementing a fair nondeterministic choice and assuming the correct functioning of processors and communication media, the authors propose an efficient transactional implementation of the atomic rendezvous in the presence of processor failures in a multiprocessor machine. Both atomicity and efficiency are obtained by using high-speed stable storage devices.>