Atmospheric composition is an essential part of weather, climate and Earth system modeling. However, modeling atmospheric composition is a computationally expensive and time-consuming task that requires a significant amount of energy. As models scale to finer spatial and temporal resolutions, maintaining real-time performance becomes increasingly challenging. To address this, optimization and acceleration techniques are essential. One promising approach is the use of deep neural networks, which have demonstrated the capability to efficiently approximate complex systems with high accuracy. Predictions using these neural networks are notably faster compared to traditional methods, significantly reducing the computational burden. In this study, we present the development of a surrogate model designed to emulate ISORROPIA, a traditional model used for calculating the concentrations of chemical compounds in the ICON-ART (ICOsahedral Nonhydrostatic model with Aerosol and Reactive Trace gases) model. Specifically, ISORROPIA is an aerosol thermodynamic equilibrium model used by ART that requires substantial computational resources, occupying a significant portion of the overall calculation time, making it particularly well-suited for emulation. The methodology involved generating a comprehensive dataset using the traditional model, which served as the training data for the neural network. This dataset encompassed a wide range of chemical concentrations and conditions, ensuring the neural network could effectively learn the underlying patterns and relationships for real-life scenarios. A simple feedforward architecture was used and fine-tuned with the primary goal of maintaining a low approximation error while also striving to achieve the lowest possible inference timing. After training, the new neural network model was compared to ISORROPIA on ICON-ART simulation data. The results demonstrated that the neural network model successfully achieved the desired outcomes, maintaining low approximation error across the globe and efficient inference timing.

Sparse linear algebra is a cornerstone of many scientific computing and machine learning applications. Python has become a popular choice for these applications due to its simplicity and ease of use. Yet high-performance sparse kernels in Python remain limited in functionality, especially on modern CPU and GPU architectures. We present pyGinkgo, a lightweight and Pythonic interface to the Ginkgo library, offering high-performance sparse linear algebra support with platform portability across CUDA, HIP, and OpenMP backends. pyGinkgo bridges the gap between high-performance C++ backends and Python usability by exposing Ginkgo’s capabilities via Pybind11 and a NumPy and PyTorch compatible interface. We benchmark pyGinkgo’s performance against state-of-the-art Python libraries including SciPy, CuPy, PyTorch and TensorFlow. Results across hardware from different vendors demonstrate that pyGinkgo consistently outperforms existing Python tools in both Sparse Matrix Vector (SpMV) product and iterative solver performance, while maintaining performance parity with native Ginkgo C++ code. Our work positions pyGinkgo as a compelling backend for sparse machine learning models and scientific workflows.

The gradients used to train neural networks are typically computed using backpropagation. While an efficient way to obtain exact gradients, backpropagation is computationally expensive, hinders parallelization, and is biologically implausible. Forward gradients are an approach to approximate the gradients from directional derivatives along random tangents computed by forward-mode automatic differentiation. So far, research has focused on using a single tangent per step. This paper provides an in-depth analysis of multi-tangent forward gradients and introduces an improved approach to combining the forward gradients from multiple tangents based on orthogonal projections. We demonstrate that increasing the number of tangents improves both approximation quality and optimization performance across various tasks.

The FAIR principles are globally accepted guidelines for improved data management practices with the potential to align data spaces on a global scale. In practice, this is only marginally achieved through the different ways in which organizations interpret and implement these principles. The concept of FAIR Digital Objects provides a way to realize a domain-independent abstraction layer that could solve this problem, but its specifications are currently diverse, contradictory, and restricted to semantic models. In this work, we introduce a rigorously formalized data model with a set of assertions using formal expressions to provide a common baseline for the implementation of FAIR Digital Objects. The model defines how these objects enable machine-actionable decisions based on the principles of abstraction, encapsulation, and entity relationship to fulfill FAIR criteria for the digital resources they represent. We provide implementation examples in the context of two use cases and explain how our model can facilitate the (re)use of data across domains. We also compare how our model assertions are met by FAIR Digital Objects as they have been described in other projects. Finally, we discuss our results’ adoption criteria, limitations, and perspectives in the big data context. Overall, our work represents an important milestone for various communities working towards globally aligned data spaces through FAIRification.

This study explores the learning dynamics of neural networks by analyzing the singular value decomposition (SVD) of their weights throughout training. Our investigation reveals that an orthogonal basis within each multidimensional weight’s SVD representation stabilizes during training. Building upon this, we introduce Orthogonality-Informed Adaptive Low-Rank (OIALR) training, a novel training method exploiting the intrinsic orthogonality of neural networks. OIALR seamlessly integrates into existing training workflows with minimal accuracy loss, as demonstrated by benchmarking on various datasets and well-established network architectures. With appropriate hyperparameter tuning, OIALR can surpass conventional training setups, including those of state-of-the-art models.

In the last decade, deep learning (DL) has significantly impacted industry and science. Initially largely motivated by computer vision tasks in 2-D imagery, the focus has shifted toward 3-D data analysis. In particular, 3-D surface reconstruction, i.e., reconstructing a 3-D shape from sparse input, is of great interest to a large variety of application fields. DL-based approaches show promising quantitative and qualitative surface reconstruction performance compared to traditional computer vision and geometric algorithms. This survey provides a comprehensive overview of these DL-based methods for 3-D surface reconstruction. To this end, we will first discuss input data modalities, such as volumetric data, point clouds, and RGB, single-view, multiview, and depth images, along with corresponding acquisition technologies and common benchmark datasets. For practical purposes, we also discuss evaluation metrics enabling us to judge the reconstructive performance of different methods. The main part of the document will introduce a methodological taxonomy ranging from point-and mesh-based techniques to volumetric and implicit neural approaches. Recent research trends, both methodological and for applications, are highlighted, pointing toward future developments.

Stencil codes are valuable methods to solve partial differential equations of models in a wide range of applications in science and engineering. Graphics processing units (GPUs) provide a highly parallel architecture with fast directly accessible memory that is desirable to run stencil codes. They enable larger and more complex simulations that are solved faster compared to simulating on CPUs. We provide a solution for users to run highly parallel stencil codes on GPUs, that can also be efficiently used on a distributed GPU cluster.In this work, we present an extension to the multi-disciplinary framework NAStJA originally designed to efficiently run stencil codes on the CPUs in current high-performance computing (HPC) systems, to also be able to run on GPUs. We describe different methods to increase the performance of stencil codes on GPUs like a border exchange method which is transparent to the user, as well as a buffer for gradient values that are needed multiple times. We show their performance using the phase-field method as a stencil code example.With this GPU extension and optimizations implemented into the NAStJA framework, we can show highly improved performance compared to the CPU implementation, and an efficiency of 92% (weak scaling) for a large-scale example simulation on 64 GPUs on the ForHLR II HPC system.

To cope with the rapid growth in available data, the efficiency of data analysis and machine learning libraries has recently received increased attention. Although great advancements have been made in traditional array-based computations, most are limited by the resources available on a single computation node. Consequently, novel approaches must be made to exploit distributed resources, e.g. distributed memory architectures. To this end, we introduce HeAT, an array-based numerical programming framework for large-scale parallel processing with an easy-to-use NumPy-like API. HeAT utilizes PyTorch as a node-local eager execution engine and distributes the workload on arbitrarily large high-performance computing systems via MPI. It provides both low-level array computations, as well as assorted higher-level algorithms. With HeAT, it is possible for a NumPy user to take full advantage of their available resources, significantly lowering the barrier to distributed data analysis. When compared to similar frameworks, HeAT achieves speedups of up to two orders of magnitude.

When training a classifier the choice of loss function heavily influences the characteristics of the resulting model. The most commonly used loss function for classification is cross entropy. In image segmentation problems where each pixel is assigned to a particular class, overlap-based losses have recently been shown to improve classifier performance especially for datasets with an imbalanced class distribution. This is particu-larly relevant to segmentation because class imbalance mitigation strategies used in regular classification are often not applicable. Overlap-based losses, however, have different drawbacks. We are aiming at combining the upsides of different losses with a simple scheduling scheme during training while minimizing their downsides. Gradually transitioning from an overlap-based dice loss to cross entropy allows to reliably select a distinct minimum in the optimization landscape as a valuable alternative to results obtained from traditional unscheduled loss functions. We demonstrate the efficacy of our approach on different combinations of loss functions, datasets, and models.

The increase in compute power and development of sophisticated simulation models with higher resolution output triggers a need for compression algorithms for scientific data. Several compression algorithms are currently under development. Most of these algorithms are using prediction-based compression algorithms, where each value is predicted and the residual between the prediction and true value is saved on disk. Currently there are two established forms of residual calculation: Exclusive-or and numerical difference. In this paper we will summarize both techniques and show their strengths and weaknesses. We will show that shifting the prediction and true value to a binary number with certain properties results in a better compression factor with minimal additional computational costs. This gain in compression factor allows for the usage of less sophisticated prediction algorithms to achieve a higher throughput during compression and decompression. In addition, we will introduce a new encoding scheme to achieve an 9% increase in compression factor on average compared to the current state-of-the-art.

One of the scientific communities that generate the largest amounts of data today are the climate sciences. New climate models enable model integration at unprecedented resolution, simulating decades and centuries of climate change, including many complex interactions in the Earth system, under different scenarios. Previously, the CPU intensive numerical integration’s used to be the bottleneck. Nowadays, limited storage space and ever increasing model output is the bigger challenge. The number of variables stored for post-processing analysis has to be limited to keep the data amounts small. For this reason, we look at lossless compression of climate data to make better use of available storage space. More specifically, we investigate prediction-based data compression. In prediction-based compression, data is processed in a predefined sequence. A prediction is provided for each data point based on prior data in the sequence. We show that there is a significant dependence of the compression ratio on the chosen traversal method and the underlying spatiotemporal data model. We examine the influence of this structural dependency on compression algorithms and explore possibilities to retrieve this information to improve compression ratios. To do this, we introduce the concept of Information Spaces (IS), which helps improve the predictions made by individual predictors by nearly 10% on average. More importantly, the standard deviation of the compression results is decreased by over 20% on average. The use of IS provides better predictions and more consistent compression ratios. Furthermore, it allows options for consolidation and fine-granular tuning of predictions, which are not possible with many common approaches used today.

Digital data of patients can aid a pathologist along the diagnostic process [1]. Medical devices generate data sets that are processed by specialized computing applications, which often run on a single computer. The resolution power of the devices is increasing steadily and, consequently, the volumes of the data sets are also growing and can no longer be analyzed in a reasonable amount of time. Big Data tools like Apache Spark [2] provide methods for analyzing data, however, they are not directly applicable and need considerable implementation efforts, in general. Usually, well established analysis tools for medical data are designed to run on single workstations. These tools are not designed to meet current and future challenges. Migrating processing tools from single nodes to distributed environments is nontrivial. Moreover, partitioning data sets for a parallel processing is a further challenge [3].In this work, we continue our efforts for improving the speedup of a bio-medical big data application further by partitioning the images of a Whole Slide Image (WSI) [4] into sub–tiles and by analyzing these sub–tiles on a cluster of computer nodes. The idea is to benefit from the divide and conquer strategy. However, it is shown that the score parameter is determined incorrectly, when the software package is applied to each sub–tile and the score parameters of all sub–tiles are combined in an apparently natural manner. The cause of this anomaly is determined and a solution suggested. The original software is based on implicit assumptions. For example, the size of the tiles is assumed to be 1024 × 1024 px2. The anomaly shows up when this constraint is reduced.

Through the introduction of next-generation models the climate sciences have experienced a breakthrough in high-resolution simulations. In the past, the bottleneck was the numerical complexity of the models, nowadays it is the required storage space for the model output. One way to tackle the data storage challenge is through data compression.

Summary Applications for biomedical data processing often integrate external libraries and frameworks for common algorithmic tasks. It typically reduces development time and increases overall code quality. With the introduction of lightweight container‐based virtualization, the bundling of applications and their required dependencies has become feasible, and containers can be transferred and executed in distributed environments. However, the incorporation of unreviewed code poses a security threat as it might contain malicious components. In this paper, measures to minimize risks of untrusted application execution are presented. Based on the system calls issued during sample execution of the application, both the container itself and the container runtime configuration are restricted to the set of actions the application requires. It is shown that the employed security measures are suited to counteract different attacks while application runtime is not affected.

Applications for data analysis of biomedical data are complex programs and often consist of multiple components. Re-usage of existing solutions from external code repositories or program libraries is common in algorithm development. To ease reproducibility as well as transfer of algorithms and required components into distributed infrastructures Linux containers are increasingly used in those environments, that are at least partly connected to the internet. However concerns about the untrusted application remain and are of high interest when medical data is processed. Additionally, the portability of the containers needs to be ensured by using only security technologies, that do not require additional kernel modules. In this paper we describe measures and a solution to secure the execution of an example biomedical application for normalization of multidimensional biosignal recordings. This application, the required runtime environment and the security mechanisms are installed in a Docker-based container. A fine-grained restricted environment (sandbox) for the execution of the application and the prevention of unwanted behaviour is created inside the container. The sandbox is based on the filtering of system calls, as they are required to interact with the operating system to access potentially restricted resources e.g. the filesystem or network. Due to the low-level character of system calls, the creation of an adequate rule set for the sandbox is challenging. Therefore the presented solution includes a monitoring component to collect required data for defining the rules for the application sandbox. Performance evaluation of the application execution shows no significant impact of the resulting sandbox, while detailed monitoring may increase runtime up to over 420%.

Significant increases in computational resources have enabled the development of more complex and spatially better resolved weather and climate models. As a result the amount of output generated by data assimilation systems and by weather and climate simulations is rapidly increasing e.g. due to higher spatial resolution, more realisations and higher frequency data. However, while compute performance has increased significantly because of better scaling program code and increasing number of cores the storage capacity is only increasing slowly. One way to tackle the data storage problem is data compression. Here, we build the groundwork for an environmental data compressor by improving compression for established weather and climate indices like El Niño Southern Oscillation (ENSO), North Atlantic Oscillation (NAO) and Quasi-Biennial Oscillation (QBO). We investigate options for compressing these indices by using a statistical method based on the Auto Regressive Integrated Moving Average (ARIMA) model. The introduced adaptive approach shows that it is possible to improve accuracy of lossily compressed data by applying an adaptive compression method which preserves selected data with higher precision. Our analysis reveals no potential for lossless compression of these indices. However, as the ARIMA model is able to capture all relevant temporal variability, lossless compression is not necessary and lossy compression is acceptable. The reconstruction based on the lossily compressed data can reproduce the chosen indices to such a high degree that statistically relevant information needed for describing climate dynamics is preserved. The performance of the (seasonal) ARIMA model was tested with daily and monthly indices.

We present a distributed system for storage, processing, three-dimensional visualisation and basic analysis of data from Earth-observing satellites. The database and the server have been designed for high performance and scalability, whereas the client is highly portable thanks to having been designed as a HTML5- and WebGL-based Web application. The system is based on the so-called MEAN stack, a modern replacement for LAMP which has steadily been gaining traction among high-performance Web applications. We demonstrate the performance of the system from the perspective of an user operating the client.

A data center is often also a Cloud center, which delivers its computational and storage capacity as services. To enable on-demand resource provision with elasticity and high reliability, the host machines in data centers are usually virtualized, which brings a challenging research topic, i.e., how to schedule the virtual machines (VM) on the hosts for energy efficiency. The goal of this Work is to ameliorate, through scheduling, the energy-efficiency of data center. To support this work a novel VM scheduling mechanism design and implementation will be proposed. This mechanism addresses on both load-balancing and temperature-awareness with a final goal of reducing the energy consumption of a data centre. Our scheduling scheme selects a physical machine to host a virtual machine based on the user requirements, the load on the hosts and the temperature of the hosts, while maintaining the quality of the service. The proposed scheduling mechanism on CloudSim will be finally validated, a well-known simulator that models data centers provisioning Infrastructure as a Service. For a comparative study, we also implemented other scheduling algorithms i.e., non power control, DVFS and power aware ThrMu. The experimental results show that the proposed scheduling scheme, combining the power-aware with the thermal-aware scheduling strategies, significantly reduces the energy consumption of a given Data Center because of its thermal-aware strategy and the support of VM migration mechanisms.

Considering the wide usage of databases and their ever growing size, it is crucial to improve the query processing performance. Selection of an appropriate set of indexes for the workload processed by the database system is an important part of physical design and performance tuning. This selection is a non-trivial tasks, especially considering possible number of native indexes in modern databases. We introduce a new approach to the index selection problem using data mining. The method recommends the creation of indexes as well as the type of each index. This results in more precise index recommendations that allows not only to create ascending and descending indexes, but also special indexes supported by the database system. Mining of queries results in candidate indexes for which virtual indexes get created. As the approach does not require modifications of the database system, it is generically applicable. Evaluations of the scalability are given for different workloads for the document-based NoSQL database MongoDB.

With the number of different cloud providers growing the importance for reliable and up-to-date performance data increases to make qualified decision on what providers to use to carry out different tasks. To cope with this issue, this paper proposes a framework usable for distributed and self-organized continuous benchmarking of hybrid and heterogeneous cloud environments. The framework implements a multipurpose job scheduling mechanism spread over a definable set of data centres connected via the open-sourced XMPP protocol. To increase scalability the system's network structure reflects physical geographical sites of the data centres.

As the data needs in every field continue to grow, storage systems have to grow and therefore need to adapt to the increasing demands of performance, reliability and fault tolerance. This also increases their complexity and costs. Improving the performance and scalability of storage systems while maintaining low costs is thus crucial. The evaluated open source storage system Ceph promises to reliably store data distributed across many nodes. Ceph is targeted at commodity hardware. This study investigates how Ceph performs in different setups and compares this with the theoretical maximum performance of the hardware. We used a bottom-up approach to benchmark Ceph at different architectural levels. We varied the amount of storage nodes and clients to test the scalability of the system. Our experiments revealed that Ceph delivers the promised scalability, and uncovered several points with improvement potential. We observed a significant increase of the write throughput by moving the Ceph journal to a faster location (in memory). Moreover, while the system scaled with the increasing number of clients operating the cluster, we noticed a slight performance degradation after the saturation point. We tested two optimisation strategies - increasing the available RAM or the object size - and noted a write throughput increase of up to 9% and 27%, respectively. Our findings improve the understanding of Ceph and should benefit future users through the presented strategies for tackling various performance limitations.

Cloud computing popularity is growing rapidly and consequently the number of companies offering their services in the form of Software-as-a-Service (SaaS) or Infrastructure-as-a-Service (IaaS) is increasing. The diversity and usage benefits of IaaS offers are encouraging SaaS providers to lease resources from the Cloud instead of operating their own data centers. However, the question remains for them how to, on the one hand, exploit Cloud benefits to gain less maintenance overheads and on the other hand, maximize the satisfactions of customers with a wide range of requirements. The complexity of addressing these issues prevent many SaaS providers to benefit from the Cloud infrastructures. In this paper, we propose HS4MC approach for automatic service selection by considering SLA claims of SaaS providers. The novelty of our approach lies in the utilization of prospect theory for the service ranking that represents a natural choice for scoring of comparable services due to the users preferences. The HS4MC approach first constructs a set of SLAs based on the given accumulated SaaS provider requirements. Then, it selects a set of services that best fulfills the SLAs. We evaluate our approach in a simulated environment by comparing it with a state-of-the-art utility- based algorithm. The evaluation results show that our approach selects services that more effectively satisfy the SLAs.

Large scale applications are emerged as one of the important applications in distributed computing. Today, the economic and technical benefits offered by the Cloud computing technology encouraged many users to migrate their applications to Cloud. On the other hand, the variety of the existing Clouds requires them to make decisions about which providers to choose in order to achieve the expected performance and service quality while keeping the payment low. In this paper, we present a multi-dimensional resource allocation scheme to automate the deployment of data-intensive large scale applications in Mutli-Cloud environments. The scheme applies a two level approach in which the target Clouds are matched with respect to the Service Level Agreement (SLA) requirements and user payment at first and then the application workloads are distributed to the selected Clouds using a data locality driven scheduling policy. Using an implemented Multi-Cloud simulation environment, we evaluated our approach with a real data-intensive workflow application in different scenarios. The experimental results demonstrate the effectiveness of the implemented matching and scheduling policies in improving the workflow execution performance and reducing the amount and costs of Intercloud data transfers.

Since Cloud services are billed by the pay-as-you-go principle, organizations can save huge investment costs. Hence, they want to know, what costs will arise by the usage of those services. On the other hand, Cloud providers want to provide the best-matching hardware configurations for different use-cases. Therefore, CloudSim, a popular event-based framework, was developed to model and simulate the usage of IaaS (Infrastructure-as-a-Service) Clouds. Metrics like usage costs, resource utilization and energy consumption can be also investigated using CloudSim. But this favored simulation framework does not provide any mechanisms to simulate todays object storage based Cloud-services (STaaS, Storage-as-a-Service). In this paper, we propose a storage extension for CloudSim to enable the simulations of STaaS-components. Interactions between users and the modeled STaaS Clouds are inspired by the CDMI (Cloud Data Management Interface) standard. In order to validate our extension, we evaluated the resource utilization and costs that arise by the usage of STaaS Clouds based on different simulation scenarios.

Analyses in climate research typically operate on large datasets stored in file hierarchies. However, e.g. Indexing, meta-data search and replication is challenging with this approach. A new approach is to store and index datasets in large, distributed databases. In order to demonstrate the performance improvement, we utilized the so-called general matching problem between measurements of two satellites that differ in orbits and measurement cycles. For comparison purpose the measurements are matched within a specified maximum spatial and time offset constraint. We describe the steps from a single-threaded approach using a SQL database to a multi-threaded approach using the NoSQL database MongoDB. The performance as well as limitations on CPU and I/O are evaluated and discussed for each approach. The performance has been improved up to a factor of 46 using 11 threads on a 12-core system.

In this paper, the method of data intensive computing is studied for large amounts of data in computed tomography (CT). An automatic workflow is built up to connect the tomography beamline of ANKA with the large scale data facility (LSDF), able to enhance the data storage and analysis efficiency. In this workflow, this paper focuses on the parallel computing of 3D computed tomography reconstruction. Different from the existing reconstruction system with filtered back-projection method, an algebraic reconstruction technique based on compressive sampling theory is presented to reconstruct the data from ultrafast computed tomography with fewer projections. Then the connected computing resources at the LSDF are used to implement the 3D CT reconstruction by distributing the whole job into multiple tasks executed in parallel. Promising reconstruction images and high computing performance are reported. For the 3D X-ray CT reconstruction, less than six minutes are actually required. LSDF is not only able to organize data efficiently, but also can provide reconstructed results to users in nearly instantaneous time. After integration into the workflow, this data intensive computing method will largely improve the data processing for ultrafast computed tomography at ANKA.

To cope with the growing requirements of data intensive scientific experiments, models and simulations the Large Scale Data Facility (LSDF) at KIT aims to support many scientific disciplines. The LSDF is a distributed storage facility at Exabyte scale providing storage, archives, data bases and meta data repositories. Apart from data storage many scientific communities need to perform data processing operations as well. For this purpose the LSDF Execution Framework for Data Intensive Applications (LAMBDA) was developed to allow asynchronous high-performance data processing next to the LSDF. However, it is not restricted to the LSDF or any special feature only available at the LSDF. The main goal of LAMBDA is to simplify large scale data processing for scientific users by reducing complexity, responsibility and error-proneness. The description of an execution is realized as part of LAMBDA administration in the background via meta data that can be obtained from arbitrary sources. Thus, the scientific user has only to select which applications he wants to apply to his data.

Digital methods and collaborative research in virtual research environments are gaining in importance for the arts and humanities. The EU-funded project DARIAH aims to enhance and support digitally-enabled research across these disciplines. The most basic but nevertheless fundamental task of DARIAH is to provide sustainable storage for research data. Information contained in data like images, texts or music needs to be secured and to remain accessible even if the original information carrier becomes lost or corrupted. The heterogeneity of the humanistic data and the need for distributed, perform ant access are the main challenges in designing an archiving system for the arts and humanities. Using the "Virtual Scriptorium", a digitisation project in Trier, Germany, this paper exemplary identifies the humanistic researchers' storage needs and derives requirements for an infrastructure. As a solution, a generic architecture for a federated data zone based on the iRODS technologies is proposed. The system implemented in Trier and Karlsruhe is described and will be extended to other locations as the researchers benefit from the initial set-up.

E-health makes use of information and communication methods and the latest e-research tools to support the understanding of body functions. E-scientists in this field take already advantage of one single infrastructure to perform computationally-intensive investigations of the human body that tend to consider each of the constituent parts separately without taking into account the multiple important interactions between them. But these important interactions imply an increasing complexity of applications that embrace multiple physical models (i.e. multi-physics) and consider a larger range of scales (i.e. multi-scale) thus creating a steadily growing demand for interoperable infrastructures that allow for new innovative application types of jointly using different infrastructures for one application. But interoperable infrastructures are still not seamlessly provided and we argue that this is due to the absence of a realistically implementable infrastructure interoperability reference model that is based on lessons learned from e-science usage. Therefore, the goal of this paper is to explore the potential of using multiple infrastructures for one scientific goal with a particular focus on e-health. Since e-scientists gain more interest in using multiple infrastructures there is a clear demand for interoperability between them to enable a use with one e-research tool. The paper highlights work in the context of an e-Health blood flow application while the reference model is applicable to other e-science applications as well.

In the last decade, life science applications have become more and more integrated into e-Science environments, hence they are typically very demanding, both in terms of computational capabilities and data capacities. Especially the access to life science applications, embedded in such environments via Grid clients still constitutes a major hurdle for scientists that do not have an IT background. Life science applications often comprise a whole set of small programs instead of a single executable. Many of the graphical Grid clients are not perfectly suited for these types of applications, as they often assume that Grid jobs will run a single executable instead of a set of chained executions (i.e. sequences). This means that in order to execute a sequence of multiple programs on a single Grid resource, piping data from one program to the next, the user would have to run a hand-written shell script. Otherwise each program is independently scheduled as a Grid job, which causes unnecessary file transfers between the jobs, even if they are scheduled on the same resource. We present a generic solution to this problem and provide a reference implementation, which seamlessly integrates with the Grid middleware UNICORE. Our approach focuses on a comfortable user interface for the creation of such program sequences, validated in UNICORE-driven HPC-based Grids. Thus, we applied our approach in order to provide support for the usage of the AMBER package (a widely-used collection of programs for molecular dynamics simulations) within Grid workflows. We finally provide a scientific use case of our approach leveraging the interoperability of two different scientific infrastructures that represents an instance of the infrastructure interoperability reference model.

Abstract Many production Grid and e‐Science infrastructures have begun to offer services to end‐users during the past several years with an increasing number of scientific applications that require access to a wide variety of resources and services in multiple Grids. Therefore, the Grid Interoperation Now—Community Group of the Open Grid Forum—organizes and manages interoperation efforts among those production Grid infrastructures to reach the goal of a world‐wide Grid vision on a technical level in the near future. This contribution highlights fundamental approaches of the group and discusses open standards in the context of production e‐Science infrastructures. Copyright © 2009 John Wiley & Sons, Ltd.

Simulation and thus scientific computing is the third pillar alongside theory and experiment in todays science and engineering. The term e-science evolved as a new research field that focuses on collaboration in key areas of science using next generation infrastructures to extend the powers of scientific computing. This paper contributes to the field of e-science as a study of how scientists actually work within currently existing Grid and e-science infrastructures. Alongside numerous different scientific applications, we identified several common approaches with similar characteristics in different domains. These approaches are described together with a classification on how to perform e-science in next generation infrastructures. The paper is thus a survey paper which provides an overview of the e-science research domain.

In a distributed grid environment with ambitious service demands the job submission and management interfaces provide functionality of major importance. Emerging e-science and grid infrastructures such as EGEE and DEISA rely on highly available services that are capable of managing scientific jobs. It is the adoption of emerging open standard interfaces which allows the distribution of grid resources in such a way that their actual service implementation or grid technologies are not isolated from each other, especially when these resources are deployed in different e-science infrastructures that consist of different types of computational resources. This paper motivates the interoperability of these infrastructures and discusses solutions. We describe the adoption of various open standards that recently emerged from the open grid forum (OGF) in the field of job submission and management by well-known grid technologies, respectively gLite and UNICORE. This has a fundamental impact on the interoperability between these technologies and thus within the next generation e-science infrastructures that rely on these technologies.

In the past several years, many scientific applications from various domains have taken advantage of e-science infrastructures that share storage or computational resources such as supercomputers, clusters or PC server farms across multiple organizations. Especially within e-science infrastructures driven by high-performance computing (HPC) such as DEISA, online visualization and computational steering (COVS) has become an important technique to save compute time on shared resources by dynamically steering the parameters of a parallel simulation. This paper argues that future supercomputers in the Petaflop/s performance range with up to 1 million CPUs will create an even stronger demand for seamless computational steering technologies. We discuss upcoming challenges for the development of scalable HPC applications and limits of future storage/IO technologies in the context of next generation e- science infrastructures and outline potential solutions.

Many existing Grid technologies and resource management systems lack a standardized job submission interface in Grid environments or e-Infrastructures. Even if the same language for job description is used, often the interface for job submission is also different in each of these technologies. The evolvement of the standardized Job Submission and Description Language (JSDL) as well as the OGSA - Basic Execution Services (OGSA-BES) pave the way to improve the interoperability of all these technologies enabling cross-Grid job submission and better resource management capabilities. In addition, the BytelO standards provide useful mechanisms for data access that can be used in conjunction with these improved resource management capabilities. This paper describes the integration of these standards into the recently released UNICORE 6 Grid middleware that is based on open standards such as the Web Services Resource Framework (WS-RF) and WS-Addressing (WS-A).

The UNICORE grid technology provides a seamless, secure and intuitive access to distributed grid resources such as computational or storage related resources. In addition, its extensible character through application-specific plug-ins and its enhancements developed in various European-funded projects leads to the UNICORE technology that is used in daily production at many supercomputer centers and research facilities world-wide today. In this paper we present an enhancement that provides the dynamic capabilities of a secure shell terminal within the UNICORE grid technology while single sign-on remains. This enhancement allows the integration of the dynamic work-behavior of scientists, or existing scientific applications, to be more integrated into the usual workflow with UNICORE and therefore in collaborative grid environments. As a well-known tool in the scientific community, a secure shell terminal provides the most flexible way of working on remote systems that no graphical user interface or advanced tooling in grid computing can ever provide

Summary form only given. We present a comparison of CPLEX-computed job schedules with the self-tuning dynP scheduler. This scheduler switches the active scheduling policy dynamically during run time, in order to reject changing characteristics of waiting jobs. Each times the self-tuning dynP scheduler checks for a new policy a quasi offline scheduling is done as the numbers of jobs are fixed. Two questions arise from this fact: what is the optimal schedule in each self-tuning step? And what is the performance difference between the optimal schedule and the best schedule generated with one of the scheduling policies? For that we model the scheduling problem as an integer problem, which is then solved with the well-known CPLEX library. Due to the size of the problem, we apply time-scaling, i.e. the schedule is computed on a larger than one second precise scale. We use the CTC job trace as input for a discrete event simulation and evaluate the performance difference between the CPLEX-computed schedules and the schedules generated by the self-tuning dynP scheduler. The results show, that the performance of the self-tuning dynP scheduler is close to solutions computed by CPLEX. However, the self-tuning dynP scheduler needs much less time for generating the schedules than CPLEX.

Summary form only given. We present a new decider mechanism for the self-tuning dynP job scheduler for modern resource management systems. This scheduler switches the active scheduling policy dynamically during run time, in order to reject changing characteristics of waiting jobs. The new decider explicitly prefers a single scheduling policy instead of being fair to all available policies. We use discrete event simulations to evaluate the achieved slowdown and utilization and compare the results with the fair decider mechanism and the static basic scheduling policies. The evaluation shows that the self-tuning dynP scheduler in combination with the preferred decider achieves good results and that it is superior to common static scheduling approaches, which use only a single policy.

This paper addresses the potential benefit of sharing jobs between independent sites in a grid computing environment. Also the aspect of parallel multi-site job execution on different sites is discussed. To this end, various scheduling algorithms have been simulated for several machine configurations with different workloads which have been derived from real traces. The results showed that a significant improvement in terms of a smaller average response time is achievable. The usage of multi-site applications can additionally improve the results as long as the increase of the execution time due to communication overhead is limited to about 25%.

The Testbed and Applications working group of the European Grid Forum (EGrid) is actively building and experimenting with a grid infrastructure connecting several research-based supercomputing sites located in Europe. The paper reports on our first feasibility study: running a self-migrating version of the Cactus simulation code across the European grid testbed, including "live" remote data visualization and steering from different demonstration booths at Supercomputing 2000, in Dallas, TX. We report on the problems that had to be resolved for this endeavour and identify open research challenges for building production-grade grid environments.

Presents a robust software infrastructure for metacomputing. The system is intended to be used by others as a building block for large and powerful computational grids. Much effort has been taken to develop a fault-tolerant architecture that does not exhibit a single point of failure. Furthermore, we have designed the system to be modular, lean and portable. It is available as open source code and has been successfully compiled on POSIX- and Microsoft Windows-compliant platforms. The system does not originate from a laboratory environment but has proven its robustness within two large metacomputing installations. It embodies a modular concept which allows easy integration of new or modified components. Hence, it is not necessary to buy into the system as whole. We rather encourage others to use only those components that fit into their specific environments.