Summary Predicting resource consumption and run time of computational workloads is crucial for efficient resource allocation, or cost and energy optimization. In this paper, we evaluate various machine learning techniques to predict the execution time of computational jobs. For experiments we use datasets from two application areas: scientific workflow management and data processing in the ALICE experiment at CERN. We apply a two‐stage prediction method and evaluate its performance. Other evaluated aspects include: (1) comparing performance of global (per‐workflow) versus specialized (per‐job) models; (2) impact of prediction granularity in the first stage of the two‐stage method; (3) using various feature sets, feature selection, and feature importance analysis; (4) applying symbolic regression in addition to classical regressors. Our results provide new valuable insights on using machine learning techniques to predict the runtime behavior of computational jobs.

Computer simulations with isogeometric analysis (IGA) have multiple applications, from phase-field modeling to tumor-growth simulations. We focus on the alternating-directions solver (ADS) algorithm, in which the matrix equation representing a computational problem is decomposed into parallel tasks following the binary and balanced structure of an elimination tree. In this paper, we explore the possibility of running large-scale IGA simulations using linear computational cost alternating direction solvers on top of modern data-parallel cloud computing frameworks. To this end, we propose a new way of decomposition of the elimination tree which makes the IGA alternating-direction solver effectively a large graph problem suitable for modern cloud-computing frameworks. On this basis, we propose a new algorithm for isogeometric analysis alternating-directions solver based on the Pregel computational model, used for large-scale graph-processing in the cloud. We implement a cloud-native solver using this algorithm in the Apache Giraph framework, and show that it can be applied for solution of challenging higher-order PDEs. We evaluate the solver in terms of various scalability models and run configurations. The results indicate linear scalability of the proposed algorithm with respect to the number of elements in the mesh.

Source code comprehension of massively growing code bases is the first crucial step in today’s software development. One of the significant obstacles in reading the source code is understanding the code dependencies. To aid this process, we developed Graph Buddy, a tool integrated as a plugin for popular IDEs, which enables interactive visual code browsing as a graph of semantic code dependencies. To preserve the direct relation to the source code, visualize, and process only the right amount of details, we have developed Semantic Code Graph (SCG), an information model underlying Graph Buddy, capable of representing detailed code semantics. We implemented the SCG model extraction for the Scala and Java languages. To evaluate Graph Buddy, we conducted a user survey on a group of 10 programmers who used the tool in three programming-related tasks. The survey indicates that the tasks would be difficult to solve without the visual browsing tool. We believe that the proposed solution has a strong potential to enhance the way programmers comprehend code dependencies.

The increasing popularity of the serverless computing approach has led to the emergence of new cloud infrastructures working in Container-as-a-Service (CaaS) model like AWS Fargate, Google Cloud Run, or Azure Container Instances. New infrastructures facilitate an innovative approach to running cloud containers where developers are freed from managing underlying resources. In this paper, we focus on evaluating the capabilities of elastic containers and their usefulness for scientific computing in the scientific workflow paradigm using AWS Fargate and Google Cloud Run infrastructures. For the experimental evaluation of our approach, we extended the HyperFlow engine to support these CaaS platforms, together with adapting four scientific workflows composed of several dozen to hundreds of tasks organized into a dependency graph. Studied applications are used to create cost-performance benchmarks and flow execution plots, delay, elasticity, and scalability measurements. Results show that serverless containers can be successfully utilized for running scientific workflows. Moreover, the results allow for gaining insight into the specific advantages and limits of the studied platforms.

Serverless computing, also known as Function-as-a-Service (FaaS) or Cloud Functions, is a new method of running distributed applications by executing functions on the infrastructure of cloud providers. Although it frees the developers from managing servers, there are still decisions to be made regarding selection of function configurations based on the desired performance and cost. The billing model of this approach considers time of execution, measured in 100ms units, as well as the size of the memory allocated per function. In this paper, we look into the problem of scheduling scientific workflows, which are applications consisting of multiple tasks connected into a dependency graph. We discuss challenges related to workflow scheduling and propose the Serverless Deadline-Budget Workflow Scheduling (SDBWS) algorithm adapted to serverless platforms. We present preliminary experiments with a small-scale Montage workflow run on the AWS Lambda infrastructure.

Early Warning Systems (EWS) may become a powerful tool for mitigating the negative impact of natural disasters, especially when combined with advanced IT solutions - such as on-demand scenario simulations, semi-automatic impact assessment, or real-time analysis of measurements from in-situ sensors. However, such complex systems require a proper computing environment supporting their development and operation. We propose the Common Information Space (CIS), a software framework facilitating design, deployment and execution of early warning systems based on real-time monitoring of natural phenomena and computationally intensive, time-critical computations. CIS provides a service-oriented technology stack which helps design 'blueprints' for early warning application scenarios and deploy these blueprints as services - system factories enabling users to rapidly deploy new EWSs in new settings. CIS also provides advanced runtime services and resource orchestration capabilities in order to address the specific requirements of EWSs: continuous operation, highly variable resource demands and mission-critical computations. The CIS concept is validated through the Flood Early Warning System whose goal is to monitor embankments in urban areas and assist in rapid decision making whenever a dike failure results in a flooding threat.

The ViroLab Virtual Laboratory is a collaborative platform for scientists representing multiple fields of expertise while working together on common scientific goals. This environment makes it possible to combine efforts of computer scientists, virology and epidemiology experts and experienced physicians to support future advances in HIV-related research and treatment. The paper explains the challenges involved in building a modern, inter-organizational platform to support science and gives an overview of solutions to these challenges. Examples of real-world problems applied in the presented environment are also described to prove the feasibility of the solution.

We present an approach to provenance tracking and querying which enables end-users to construct complex queries over provenance records. The use of ontologies for modeling provenance, data and applications enables query construction in an end-user oriented manner, i.e. by using terms of the scientific domain familiar to end users, instead of complex query languages. In addition, our ontologies contain mappings to underlying data models and sources. This allows to construct queries over provenance which additionally explore the structure of data items in the provenance records (e.g. experiment input or output data), combining requests to provenance and data repositories.

Monitoring of running scientific workflows (experiments) is not only important for observing their execution status, but also for collecting provenance, improving performance, knowledge extraction, etc. We propose an ontology model of experiment information which describes the execution of an experiment using a well-defined semantics, and aggregates various aspects of workflow execution including provenance, performance, resource information, and others. Such multi-aspect semantic-rich information is indispensable to build knowledge services on top of it. We describe a grid workflow monitoring architecture which is necessary to collect and correlate workflow monitoring data. The process of aggregation of monitoring data into experiment information is presented. Our approach is validated on a drug resistance ranking application running in the ViroLab virtual laboratory for infectious diseases.

We propose an end-user oriented approach to querying repositories of data and provenance in e-Science environments. The approach is based on ontology models describing multiple domains - in silico experiments, provenance, data, and applications. Those ontologies, integrated in a unified model and containing mappings to underlying data models, allow to query repositories of data and provenance in a unified way, or even combine provenance and data aspects in one query. We demonstrate QUery TRanslation tools (QUaTRo), built on top of the ontology models, which allow to construct complex queries over both data and provenance repositories, expressed in the terms of the domain familiar to end users. We present, in the context of the ViroLab virtual laboratory for infectious diseases, examples of construction of complex queries, combining provenance and data model aspects, which can be of practical value to scientists or medical users.

Grid workflows for e-science are complex and prone to failures. However, there is a lack of performance monitoring and analysis tools for supporting the user as well as workflow middleware to monitor and understand the performance of complex interactions among Grid applications, middleware and resources involved in workflow executions. In this paper, we present a novel integrated environment which supports online performance monitoring and analysis of service-oriented workflows. Performance monitoring and analysis of Grid workflows and infrastructure is conducted through a Web portal. Performance overheads of Grid workflows are analyzed in a systematic way, and performance problems can be detected during runtime. Moreover, we present several languages that alleviate the interaction among performance monitoring and analysis services and their clients. Our system has been integrated into the K-WfGrid knowledge-based workflow system. It plays a key role in supporting the user and developer to analyze their workflows and in providing performance knowledge for constructing and executing workflows.

In this paper we describe a Grid-enabled system for monitoring HLA-based applications to enable load balancing by migration of federates. The monitoring is based on the OCM-G system [2].We show how the design concepts of the OCM-G enable easy adaptation to monitoring of HLA, C++-based applications. The solution presented in this paper is transparent to the user application and does not require any changes to the original HLA RTI code. We also describe the role of monitoring the whole system for managing execution of HLA-based applications described in [25]. We discuss implementation issues and present test results for the monitoring overhead.