Successful e-science requires control over variations of an experiment, typically encoded as a script or workflow, as well as the ability to transfer existing experiments to other environments in a reproducible way. Variations may be introduced either deliberately or because of inaccurate porting, for instance when the target environment does not satisfy the original dependencies on data and software libraries. Although these variations are captured by various provenance capturing systems, they were not exploited in an effective way to explain difference between two computations. In this paper we address the problem of explaining the observed differences in the outcomes from two such experiment variations, in terms of differences in the execution traces of those experiments. While experiments may differ widely in their structure and implementation, our hypothesis is that a general method for producing such explanations only needs to rely on the provenance of the experiment execution, for which using a standard data model, i.e., W3C PROV, is available. To test this hypothesis in a concrete workflow setting, we have developed why-diff, an algorithm to match two provenance traces derived from the execution of workflows that are variations of one another. We present the algorithm, show how it derives a delta graph which encodes the differences between the traces and thus provides the basis for generating human-readable explanations, and evaluate its performance in terms of the number of comparisons required during the matching process

Majority of workflows executed nowadays need to process a massive amount of data. Re-execution of such dataintensive scientific workflows often results in different outputs. Scientific research progresses when discoveries are reproduced and verified. However, simply re-enacting a scientific computation, such as a workflow, does not guarantee the correctness of results because of unintentional changes that may have interfered with the re-enactment process. We investigate the hypothesis that the metadata of a workflow execution can be used to explain why the experimenter observes different results (cause analysis). Similarly, Scientific metadata can be used to determine the impact of intentional variations that the experimenter may have injected into a new version of the workflow. We explore these two complementary cases using a simple algorithm for traversing two metadata traces in lock-step mode, which we illustrate through two human genomics data analysis workflows.

The significant growth in cloud computing has led to increasing number of cloud providers, each offering their service under different conditions - one might be more secure whilst another might be less expensive or more reliable. At the same time user applications have become more and more complex. Often, they consist of a diverse collection of software components, and need to handle variable workloads, which poses different requirements on the infrastructure. Therefore, many organisations are considering using a combination of different clouds to satisfy these needs. It raises, however, a non-trivial issue of how to select the best combination of clouds to meet the application requirements. This paper presents a novel algorithm to deploy workflow applications on federated clouds. First, we introduce an entropy-based method to quantify the most reliable workflow deployments. Second, we apply an extension of the Bell-LaPadula Multi-Level security model to address application security requirements. Finally, we optimise deployment in terms of its entropy and also its monetary cost, taking into account the cost of computing power, data storage and inter-cloud communication. We implemented our new approach and compared it against two existing scheduling algorithms: Extended Dynamic Constraint Algorithm (EDCA) and Extended Biobjective dynamic level scheduling (EBDLS). We show that our algorithm can find deployments that are of equivalent reliability but are less expensive and meet security requirements. We have validated our solution through a set of realistic scientific workflows, using well-known cloud simulation tools (WorkflowSim and DynamicCloudSim) and a realistic cloud based data analysis system (e-Science Central).

Scientific workflows are increasingly being migrated to the Cloud. However, workflow developers face the problem of which Cloud to choose and, more importantly, how to avoid vendor lock-in. This is because there are a range of Cloud platforms, each with different functionality and interfaces. In this paper we propose a solution - a system that allows workflows to be portable across a range of Clouds. This portability is achieved through a new framework for building, dynamically deploying and enacting workflows. It combines the TOSCA specification language and container-based virtualization. TOSCA is used to build a reusable and portable description of a workflow which can be automatically deployed and enacted using Docker containers. We describe a working implementation of our framework and evaluate it using a set of existing scientific workflows that illustrate the flexibility of the proposed approach.

Workflow is a well-established means by which to capture scientific methods in an abstract graph of interrelated processing tasks. The reproducibility of scientific workflows is therefore fundamental to reproducible e-Science. However, the ability to record all the required details so as to make a workflow fully reproducible is a long-standing problem that is very difficult to solve. In this paper, we introduce an approach that integrates system description, source control, container management and automatic deployment techniques to facilitate workflow reproducibility. We have developed a framework that leverages this integration to support workflow execution, re-execution and reproducibility in the cloud and in a personal computing environment. We demonstrate the effectiveness of our approach by examining various aspects of repeatability and reproducibility on real scientific workflows. The framework allows workflow and task images to be captured automatically, which improves not only repeatability but also runtime performance. It also gives workflows portability across different cloud environments. Finally, the framework can also track changes in the development of tasks and workflows to protect them from unintentional failures.

Scientific workflows play an increasingly important role in building scientific applications, while cloud computing provides on-demand access to large compute resources. Combining the two offers the potential to increase dramatically the ability to quickly extract new results from the vast amounts of scientific data now being collected. However, with the proliferation of cloud computing platforms and workflow management systems, it becomes more and more challenging to define workflows so they can reliably run in the cloud and be reused easily. This paper shows how TOSCA, a new standard for cloud service management, can be used to systematically specify the components and life cycle management of scientific workflows by mapping the basic elements of a real workflow onto entities specified by TOSCA. Ultimately, this will enable workflow definitions that are portable across clouds, resulting in the greater reusability and reproducibility of workflows.

The federation of clouds can provide benefits for cloud-based applications. Different clouds have different advantages - one might be more reliable whilst another might be more secure or less expensive. However, being able to select the best combination of clouds to meet the application requirements is not trivial. This paper presents a novel algorithm to deploy workflow applications on federated clouds. Firstly, we introduce an entropy-based method to quantify the most reliable workflow deployments. Secondly, we apply an extension of the Bell-LaPadula Multi-Level security model to meet application security requirements. Finally, we optimise deployment in terms of its entropy and also its monetary cost, taking into account the price of computing power, data storage and inter-cloud communication. To evaluate the new algorithm we compared it against two existing scheduling algorithms: Dynamic Constraint Algorithm (DCA) and Biobjective dynamic level scheduling (BDLS). We show that our algorithm can find deployments that are of equivalent reliability, but are less expensive and also meet security requirements. We have validated our solution using workflows implemented in the e-Science Central cloud-based data analysis system.

In this paper we describe our initial experiences in the Cloud-e-Genome project with moving the whole exome sequencing pipeline from the scripted HPC-based solution to a workflow enactment system running in the cloud. We discuss shortcomings of the existing approach based on scripts and list benefits that a workflow-based solution can provide. Despite the effort it involved to wrap all required tools in the form of workflow blocks and the restrictions of the dataflow model used to represent workflows we expect the migration to significantly improve the current status of the pipeline. Our target is to enable flexibility, traceability and reproducibility of the solution, so that it can better fit the evolution of tools, data and pipeline itself and allow us to run it at national scale. This work will become foundation for the more complete system that includes variant filtering and interpretation for the diagnostic purposes.

The significant increase in the use of cloud computing, has led to an interest in partitioning applications over a set of public and private clouds in order to meet a range of non-functional requirements including performance (for example where private cloud resources alone are insufficient), dependability (e.g. To allow the application to continue to operate even if one cloud fails) and security (for example to ensure that sensitive data is restricted to sufficiently secure clouds and networks). This paper describes a novel deployment planning algorithm to partition complex workflow-based applications over federated clouds, while meeting security requirements. The security issues are based on our previous work which extends the Bell-La Padula model to encompass cloud computing. Selecting the cheapest option for partitioning a workflow over a set of resources has been shown to be an NP-hard problem, which can take impractically long for partitioning large workflows over multiple clouds. We therefore introduce a novel adaptive partitioning algorithm to handle these large workflow applications, which significantly reduces the time required to choose a sufficiently good partitioning option. This is based on generating an initial partitioning, and then adapting it to see if a better solution can be found by bringing together on the same node services with significant communication costs. The algorithm has been implemented and evaluated by using both randomly generated and real world scientific workflows. The experiment results show that our algorithm is thousand times quicker than the exhaustive algorithm presented in our previous work. Yet, on average it generates only 25% more costly solutions. We also compared this algorithm with two other methods commonly used to partition workflows over a set of clouds.

The paper presents practical experiences with deployment of advanced medical teleconsultation system - TeleDICOM over public computer networks and on standard PC computers. Requirements connected with operation in interactive and non-interactive modes have been discussed. Next, the architecture of the system has been described and its influence on deployment process and upgrade procedures has been evaluated. The thick-client architecture has been chosen; in this context communication and information flows are analyzed. The performance aspects of the system are related to medical data processing on a standard PC on a client site. The existing constraints of large DICOM files processing are listed and their solution are proposed. TeleDICOM was successfully deployed in John Paul II Hospital in Krakow and in a few regional hospitals in the South of Poland what practically verifies the proposed solutions.