Abstract Scientific workflows are becoming increasingly important for complex scientific applications. Conducting real experiments for large-scale workflows is challenging because they are very expensive and time consuming. A simulation is an alternative approach to a real experiment that can help evaluating the performance of workflow management systems (WMS) and optimise workflow management techniques. Although there are several workflow simulators available today, they are often user-oriented and treat the cloud as a black box. Unfortunately, this behaviour prevents the evaluation of the infrastructure level impact of the various decisions made by the WMSs. To address these issues, we have developed a WMS simulator (called DISSECT-CF-WMS) on DISSECT-CF that exposes the internal details of cloud infrastructures. DISSECT-CF-WMS enables better energy awareness by allowing the study of schedulers for physical machines. It also enables dynamic provisioning to meet the resource needs of the workflow application while considering the provisioning delay of a VM in the cloud. We evaluated our simulation extension by running several workflow applications on a given infrastructure. The experimental results show that we can investigate different schedulers for physical machines on different numbers of virtual machines to reduce energy consumption. The experiments also show that DISSECT-CF-WMS is up to 295× faster than WorkflowSim and still provides equivalent results. The experimental results of auto-scaling show that it can optimise makespan, energy consumption and VM utilisation in contrast to static VM provisioning.

Abstract One challenge of studying cloud workload traces is the lack of available users’ identities. Therefore, clustering methods were used to address this challenge through extracting these identities from workload traces. For better extraction, it is beneficial to select attributes (columns in the traces) for clustering by using feature selection methods. However, the use of general selection methods requires details that are not available for workload traces (e.g. predefined number of clusters). Therefore, in this paper, we present an unsupervised feature selection method for cloud workload traces to identify good candidate attributes for clustering. This method uses Silhouette coefficients to rank attributes that are best for users’ extraction through clustering. The performance of our SeQual method is evaluated in comparison with commonly used (supervised and unsupervised) feature selection methods with the help of clustering quality metrics (i.e. adjusted rand index, entropy and precision). The results show that the SeQual method can compete with the supervised methods and perform better than unsupervised ones, with an average accuracy between 90% and 99%.

Users’ behaviours show a noticeable impact on cloud computing resources. Behaviour prediction models could foster usage awareness of cloud users. This requires training prediction models with datasets that provide user information. Unfortunately, such information is excluded from many relevant datasets. Therefore, in this work, we investigate the ability of extracting these identities via clustering methods. We conduct this by categorising workload datasets according to the availability of users information in their attributes. Then, we focus our attention on shared attributes between user information disclosing and non-disclosing datasets. Eventually, we evaluated the potential of several clustering approaches on user information disclosing datasets. Our results show that users’ identifications can be extracted with relatively high accuracy using clustering. They also show that the highest clustering precision is mostly obtained from the attributes representing request components that strongly relate to the user’s application.

Abstract Workload consolidation is an important method for the efficient operation of cloud data centers, impacting important quality attributes such as resource utilization and power consumption. Many different approaches have been proposed for workload consolidation, but few comparative studies were executed to date. Therefore, it is unclear which of the proposed approaches work best in which situation. In this article, we present a comprehensive simulation‐based comparison of five workload consolidation techniques. We introduce a general framework for workload consolidation techniques to the DISSECT‐CF simulator to foster the development and comparison of efficient data center consolidation algorithms. We use this framework to evaluate the effectiveness of a first fit best fit decreasing heuristic, a custom heuristic, and three population‐based metaheuristics (genetic algorithm, artificial bee colony, and particle swarm optimization). The evaluation is based on a wide variety of real‐world workload traces. The five algorithms are compared in terms of total energy consumption, the duration of the simulation, and the number of migrations. Based on the results, there is no generally best consolidation technique. The results deliver insight into the pros and cons of the algorithms as well as the impact of different parameters. In particular, the results show that population‐based metaheuristics do not offer a significant gain in terms of solution quality to compensate for the increased simulation time.

Cloud Computing has become mature enough to enable the virtualized management of multiple datacentres. Datacentre consolidation is an important method for the efficient operation of such distributed infrastructures. Several approaches have been developed to improve the efficiency e.g. in terms of power consumption, but only a few attention has been turned to combining pricing methods with consolidation techniques. In this paper we discuss how we introduced cost models to the DISSECT-CF simulator to foster the development of cost efficient datacentre consolidation solutions. We also exemplify the usage of this extended simulator by performing cost-aware datacentre consolidation. We apply real world traces to simulate cloud load, and propose 7 strategies to address the problem.

Summary The use of virtual machines (VMs) in Cloud computing provides various benefits in the overall software engineering lifecycle. These include efficient elasticity mechanisms resulting in higher resource utilization and lower operational costs. The VMs as software artifacts are created using provider‐specific templates, called virtual machine images (VMI), and are stored in proprietary or public repositories for further use. However, some technology‐specific choices can limit the interoperability among various Cloud providers and bundle the VMIs with nonessential or redundant software packages, leading to increased storage size, prolonged VMI delivery, stagnant VMI instantiation, and ultimately vendor lock‐in. To address these challenges, we present a set of novel functionalities and design approaches for efficient operation of distributed VMI repositories, specifically tailored for enabling (1) simplified creation of lightweight and size optimized VMIs tuned for specific application requirements; (2) multi‐objective VMI repository optimization; and (3) efficient reasoning mechanism to help optimizing complex VMI operations. The evaluation results confirm that the presented approaches can enable VMI size reduction by up to 55%, while trimming the image creation time by 66%. Furthermore, the repository optimization algorithms can reduce the VMI delivery time by up to 51% and cut down the storage expenses by 3%. Moreover, by implementing replication strategies, the optimization algorithms can increase the system reliability by 74%.

In order to minimise their energy use, data centre operators are constantly exploring new ways to construct computing infrastructures. As low power CPUs, exemplified by ARM-based devices, are becoming increasingly popular, there is a growing trend for the large scale deployment of low power servers in data centres. For example, recent research has shown promising results on constructing small scale data centres using Raspberry Pi (RPi) single-board computers as their building blocks. To enable larger scale experimentation and feasibility studies, cloud simulators could be utilised. Unfortunately, state-of-the-art simulators often need significant modification to include such low power devices as core data centre components. In this paper, we introduce models and extensions to estimate the behaviour of these new components in the DISSECT-CF cloud computing simulator. We show that how a RPi based cloud could be simulated with the use of the new models. We evaluate the precision and behaviour of the implemented models using a Hadoop-based application scenario executed both in real life and simulated clouds.

In Internet of Things (IoT), sensors, actuators and smart devices are connected to the Internet. Application providers combine this connectivity with novel scenarios involving cloud computing. Some require in depth analysis of the interaction between IoT devices and clouds. Research focuses on questions like how to govern such large cohort of devices (i.e., often over tens of thousands). Distributed systems simulators help in such analysis, but they are problematic to apply in this newly emerging domain. Most simulators are either too detailed (e.g., need extensive knowledge on networking), or not extensible enough to support the new scenarios. This paper introduces our attempt to show how a state of the art simulator could model generic IoT sensors. We show the fundamental properties of IoT entities represented in the simulator. Based on these properties, we present an XML based, declarative modelling language aiming at: (i) describing the behaviour of sensors and their relation to clouds, and (ii) allowing rapid prototyping of simulations. Finally, we validate the applicability of our IoT extensions in five scenarios in the field of weather forecasting.

Virtualization is a key enabling technology in Cloud computing that allows users to run multiple virtual machines (VMs) with their own application environment on top of physical hardware. It permits scaling up and down of applications by elastic on-demand provisioning of VMs in response to their variable load to achieve increased utilization efficiency at a lower operational cost, while guaranteeing the desired level of Quality of Service (QoS) to the end-users. Typically, VMs are created using provider-specific templates that are stored in proprietary repositories, leading to provider lock-in and hampering portability or simultaneous usage of multiple federated Clouds. In this context, optimization at the level of the virtual machine image is needed both by the applications and by the underlying Cloud providers for improved resource usage, operational costs, elasticity, storage use, and other desired QoS-related features. To overcome those issues, the ENTICE project researches and creates a novel VM repository and operational environment for federated Cloud infrastructures. There exists a large variety of industrial applications that can strongly benefit by the ENTICE environment. In this paper we present an interesting selection of complementary use cases that drive the definition of the essential requirements for the ENTICE environment, and more importantly, validate the introduced innovations.

Due to the difficulty of employing real data centres' infrastructure for assessing the effectiveness of energy-aware algorithms, many researchers resort on using Cloud simulators. These tools require precise and detailed models for virtualized data centres in order to deliver accurate results. In recent years, many models have been proposed, but most of them either do not consider energy consumption related to virtual machine(VM) migration or ignore some of the energy-impacting components (e.g. CPU, network, storage). In this paper, we propose a new model for data centre energy consumption that takes into account these omitted components. We implement this model in a framework that combines two Cloud simulators: GroudSim that provides the Cloud management side, and DISSECT-CF that provides the internal infrastructure side. We evaluated our model in a comprehensive set of scenarios and obtained an accuracy between 8% and 22% for instantaneous power consumption, and between 8% and 25% for energy consumption.

Summary Scientific workflow systems face new challenges when supporting Cloud computing, as the information on the state of the used infrastructures is much less detailed than before. Thus, organising virtual infrastructures in a way that not only supports the workflow execution but also optimises it for several service level objectives (e.g. maximum energy consumption limit, cost, reliability, availability) become reliant on good Cloud modelling and prediction information. While simulators were successfully aiding research on such workflow management systems, the currently available Cloud related simulation toolkits suffer from several issues (e.g. scalability and narrow scope) that hinder their applicability. To address these issues, this article introduces techniques for unifying two existing simulation toolkits by first analysing the problems with the current simulators, and then by illustrating the problems faced by workflow systems. We use for this purpose the example of the ASKALON environment, a scientific workflow composition and execution tool for cloud and grid environments. We illustrate the advantages of a workflow system with directly integrated simulation back‐end and how the unification of the selected simulators does not affect the overall workflow execution simulation performance. Copyright © 2015 John Wiley & Sons, Ltd.

Energy consumption has become a significant issue for data centres. Assessing their consumption requires precise and detailed models. In the latter years, many models have been proposed, but most of them either do not consider energy consumption related to virtual machine migration or do not consider the variation of the workload on (1) the virtual machines (VM) and (2) the physical machines hosting the VMs. In this paper, we show that omitting migration and workload variation from the models could lead to misleading consumption estimates. Then, we propose a new model for data centre energy consumption that takes into account the previously omitted model parameters and provides accurate energy consumption predictions for paravirtualised virtual machines running on homogeneous hosts. The new model's accuracy is evaluated with a comprehensive set of operational scenarios. With the use of these scenarios we present a comparative analysis of our model with similar state-of-the-art models for energy consumption of VM Migration, showing an improvement up to 24% in accuracy of prediction.

Infrastructure as a Service systems use virtual appliances to initiate virtual machines. As virtual appliances encapsulate applications and services with their support environment, their delivery is the most expensive task of the virtual machine creation. Virtual appliance delivery is a well-discussed topic in the field of cloud computing. However, for high efficiency, current techniques require the modification of the underlying IaaS systems. To target the wider adoptability of these delivery solutions, this article proposes the concept of minimal manageable virtual appliances (MMVA) that are capable of updating and configuring their virtual machines without the need to modify IaaS systems. To create MMVAs, we propose to reduce manageable virtual appliances until they become MMVAs. This research also reveals a methodology for appliance developers to incorporate MMVAs in their own appliances to enable their efficient delivery and wider adoptability. Finally, the article evaluates the positive effects of MMVAs on an already existing delivery solution: the Automated Virtual appliance creation Service (AVS). Through experimental evaluation, we present that the application of MMVAs not only increases the adoptability of a delivery solution but it also significantly improves its performance in highly dynamic systems.

Cloud Computing infrastructures have been developed as individual islands, and mostly proprietary solutions so far. However, as more and more infrastructure providers apply the technology, users face the inevitable question of using multiple infrastructures in parallel. Federated cloud management systems offer a simplified use of these infrastructures by hiding their proprietary solutions. As the infrastructure becomes more complex underneath these systems, the situations (like system failures, handling of load peaks and slopes) that users cannot easily handle, occur more and more frequently. Therefore, federations need to manage these situations autonomously without user interactions. This paper introduces a methodology to autonomously operate cloud federations by controlling their behavior with the help of knowledge management systems. Such systems do not only suggest reactive actions to comply with established Service Level Agreements (SLA) between provider and consumer, but they also find a balance between the fulfillment of established SLAs and resource consumption. The paper adopts rule-based techniques as its knowledge management solution and provides an extensible rule set for federated clouds built on top of multiple infrastructures.

Cloud Computing offers simple and cost effective outsourcing in dynamic service environments, and allows the construction of service-based applications using virtualization. By aggregating the capabilities of various IaaS cloud providers, federated clouds can be built. Managing such a distributed, heterogeneous environment requires sophisticated interoperation of adaptive coordinating components. In this paper we introduce an integrated federated management and monitoring approach that enables autonomous service provisioning in federated clouds. In this architecture, cloud brokers manage the number and the location of the utilized virtual machines for the received service requests. In order to provide seamless service executions, a state of the art monitoring solution is proposed that supports cloud selection performed by the management layer of the architecture. Our solution is able to cope with highly dynamic service executions by federating heterogeneous cloud infrastructures in a transparent and autonomous manner.

Virtual appliances store the required information to instantiate a functional Virtual Machine (VM) on Infrastructure as a Service (IaaS) cloud systems. Large appliance size obstructs IaaS systems to deliver dynamic and scalable infrastructures according to their promise. To overcome this issue, this paper offers a novel technique for virtual appliance developers to publish appliances for the dynamic environments of IaaS systems. Our solution achieves faster virtual machine instantiation by reducing the appliance size while maintaining its key functionality. The new virtual appliance optimization algorithm identifies the removable parts of the appliance. Then, it applies active fault injection to remove the identified parts. Afterward, our solution assesses the functionality of the reduced virtual appliance by applying the-appliance developer provided-validation algorithms. We also introduce a technique to parallelize the fault injection and validation phases of the algorithm. Finally, the prototype implementation of the algorithm is discussed to demonstrate the efficiency of the proposed algorithm through the optimization of two well-known virtual appliances. Results show that the algorithm significantly decreased virtual machine instantiation time and increased dynamism in IaaS systems.

Cloud Computing builds on the latest achievements of diverse research areas, such as Grid Computing, Service-oriented computing, business processes and virtualization. Managing such heterogeneous environments requires sophisticated interoperation of adaptive coordinating components. In this paper we introduce an SLA-aware Service Virtualization architecture that provides non-functional guarantees in the form of Service Level Agreements and consists of a three-layered infrastructure including agreement negotiation, service brokering and on demand deployment. In order to avoid costly SLA violations, flexible and adaptive SLA attainment strategies are used with a failure propagation approach. We demonstrate the advantages of our proposed solution with a biochemical case study in a Cloud simulation environment.

Manual deployment of the application usually requires expertise both about the underlying system and the application. Automatic service deployment can improve deployment significantly by using on-demand deployment and self-healing services. To support these features this paper describes an extension the globus workspace service. This extension includes creating virtual appliances for grid services, service deployment from a repository, and influencing the service schedules by altering execution planning services, candidate set generators or information systems.