With the increasing amount of digital data available for analysis and simulation, the class of I/O-intensive HPC workflows is fated to quickly expand, further exacerbating the performance gap between computing, memory, and storage technologies. This paper introduces CAPIO (Cross-Application Programmable I/O), a middleware capable of injecting I/O streaming capabilities into file-based workflows, improving the computation- I/O overlap without the need to change the application code. The contribution is twofold: 1) at design time, a new I/O coordination language allows users to annotate workflow data dependencies with synchronization semantics; 2) at run time, a user-space middleware automatically and transparently to the user turns a workflow batch execution into a streaming execution according to the semantics expressed in the configuration file. CAPIO has been tested on synthetic benchmarks simulating typical workflow I/O patterns and two real-world workflows. Experiments show that CAPIO reduces the execution time by 10% to 66% for data-intensive workflows that use the file system as a communication medium.

The designers of a new coordination interface enacting complex workflows have to tackle a dichotomy: choosing a language-independent or language-dependent approach. Language-independent approaches decouple workflow models from the host code’s business logic and advocate portability. Language-dependent approaches foster flexibility and performance by adopting the same host language for business and coordination code. Jupyter Notebooks, with their capability to describe both imperative and declarative code in a unique format, allow taking the best of the two approaches, maintaining a clear separation between application and coordination layers but still providing a unified interface to both aspects. We advocate the Jupyter Notebooks’ potential to express complex distributed workflows, identifying the general requirements for a Jupyter-based Workflow Management System (WMS) and introducing a proof-of-concept portable implementation working on hybrid Cloud-HPC infrastructures. As a byproduct, we extended the vanilla IPython kernel with workflow-based parallel and distributed execution capabilities. The proposed Jupyter-workflow (Jw) system is evaluated on common scenarios for High Performance Computing (HPC) and Cloud, showing its potential in lowering the barriers between prototypical Notebooks and production-ready implementations.

To achieve high performance and high energy efficiency on near-future exascale computing systems, three key technology gaps needs to be bridged. These gaps include: energy efficiency and thermal control; extreme computation efficiency via HW acceleration and new arithmetics; methods and tools for seamless integration of reconfigurable accelerators in heterogeneous HPC multi-node platforms. TEXTAROSSA aims at tackling this gap through a co-design approach to heterogeneous HPC solutions, supported by the integration and extension of HW and SW IPs, programming models and tools derived from European research.

The Grid is currently living an exciting evolution step to turn from a niche technological infrastructure to a widely used Business environment. To this aim, Workflows are playing an important role acting as the link towards the well-established Business Process Management environment. A number of Grid projects worldwide are trying to implement this link and create the Business Grid but no definitive proposal has been provided yet. In this paper, we analyse two European Grid projects, Akogrimo and NextGRID, both investigating the Business usage of Grid from architectural perspectives but each one adopting a different approach in proposing solutions. The comparative analysis we report provides interesting elements to define a unified vision of the Next Generation Grids where Workflow Models, Languages, and Engines constitute a fundamental building block. Understanding and sharing this vision can be the first step to get to an effective architectural design for Business Grids.

Semantic grid is becoming a key enabler for next generation grid and the need of supporting process description and enactment, by means of composition of multiple resources, emerged as one of the fundamental requirements. Within NextGRID project, the idea of adopting business processes (expressed as workflow policies) as architectural components in a next generation grid has been developed with the aim of providing architecture with dynamic behaviour and in teroperability. The need of a semantic workflow representation language then emerged and was developed defining an OWLS extension able to support workflow description. The resulting OWL-WS (OWL for workflow and services) ontology allows us modelling concept like abstract and concrete services and workflows according to a semantic workflow model also enabling specification of higher-order workflows and semantic service grouping. This language is being used for specifying adaptive business processes (policy) that are used as evaluation and binding mechanisms by a workflow enactment engine.