In the realm of High-Performance Computing (HPC), the disparity between I/O and processing capabilities can hinder application performance. One source of this I/O gap is the latency of the devices, especially in case of distributed filesystems (DFS). Prior solutions to minimize such negative effects (e.g. using asynchronous operations or separate I/O-Threads), have to be considered when designing an application. This work aims to execute these operations asynchronously for existing applications without necessitating modifications or even recompilation. To attain this objective, we introduce the Lazy Read approach, which alters how the operating system handles read requests. This modification enables the application to start multiple reads or do CPU work in parallel instead of waiting for completion of the read operation. For evaluation, we implemented the Lazy Read approach in the Linux kernel. We then showed that the approach adds overhead, however, multiple read operations can be executed in parallel. In our specific evaluation the maximum possible speedup was achieved, reducing the execution time by up to 50%.

Scientific workflows have received increasing interest and are used in many scientific fields to gather, analyze, and process significant amounts of data. However, their tasks are usually treated as black boxes, and their behavior remains unconsidered for resource allocations, which can lead to subpar resource allocations with typical scheduling. Although not done yet, it should be possible to observe such tasks, learn their behavior, and use this knowledge to improve future executions. As workflows and their tasks are often executed multiple times on a massive scale, even a slight improvement per execution may save hours of execution time and significant amounts of energy.To achieve this goal, we develop an innovative approach to model task executions and predict resource usage. The prediction is embedded in a feedback loop to repeatedly improve the models and to closely track workflow executions to make predictions and resource allocations accurate.

Scientific workflows consist of thousands of highly parallelized tasks executed in a distributed environment involving many components. Automatic tracing and investigation of the components’ and tasks’ performance metrics, traces, and behavior are necessary to support the end user with a level of abstraction since the large amount of data cannot be analyzed manually. The execution and monitoring of scientific workflows involves many components, the cluster infrastructure, its resource manager, the workflow, and the workflow tasks. All components in such an execution environment access different monitoring metrics and provide metrics on different abstraction levels. The combination and analysis of observed metrics from different components and their interdependencies are still widely unregarded.We specify four different monitoring layers that can serve as an architectural blueprint for the monitoring responsibilities and the interactions of components in the scientific workflow execution context. We describe the different monitoring metrics subject to the four layers and how the layers interact. Finally, we examine five state-of-the-art scientific workflow management systems (SWMS) in order to assess which steps are needed to enable our four-layer-based approach.

In the recent years, scientific workflows gained more and more popularity. In scientific workflows, tasks are typically treated as black boxes. Dealing with their complex interrelations to identify optimization potentials and bottlenecks is therefore inherently hard. The progress of a scientific workflow depends on several factors, including the available input data, the available computational power, and the I/O and network bandwidth. Here, we tackle the problem of predicting the workflow progress with very low overhead. To this end, we look at suitable formalizations for the key parameters and their interactions which are sufficiently flexible to de scribe the input da ta consumption, the computational effort and the output production of the workflow’s tasks. At the same time they allow for computationally simple and fast performance predictions, including a bottleneck analysis over the workflow runtime. A piecewise-defined bottleneck function is derived from the discrete intersections of the task models’ limiting functions. This allows to estimate potential performance gains from overcoming the bottlenecks and can be used as a basis for optimized resource allocation and workflow execution.