The accuracy of Machine Learning (ML) models is highly dependent on the hyperparameters that have to be chosen by the user before the training. However, finding the optimal set of hyperparameters is a complex process, as many different parameter combinations need to be evaluated, and obtaining the accuracy of each combination usually requires a full training run. It is therefore of great interest to reduce the computational runtime of this process. On High-Performance Computing (HPC) systems, several configurations can be evaluated in parallel to speed up this Hyperparameter Optimization (HPO). State-of-the-art HPO methods follow a bandit-based approach and build on top of successive halving, where the final performance of a combination is estimated based on a lower than fully trained fidelity performance metric and more promising combinations are assigned more resources over time. Frequently, the number of epochs is treated as a resource, letting more promising combinations train longer. Another option is to use the number of workers as a resource and directly allocate more workers to more promising configurations via data-parallel training. This article proposes a novel Resource-Adaptive Successive Doubling Algorithm (RASDA), which combines a resource-adaptive successive doubling scheme with the plain Asynchronous Successive Halving Algorithm (ASHA). Scalability of this approach is shown on up to 1,024 Graphics Processing Units (GPUs) on modern HPC systems. It is applied to different types of Neural Networks (NNs) and trained on large datasets from the Computer Vision (CV), Computational Fluid Dynamics (CFD), and Additive Manufacturing (AM) domains, where performing more than one full training run is usually infeasible. Empirical results show that RASDA outperforms ASHA by a factor of up to 1.9 with respect to the runtime. At the same time, the solution quality of final ASHA models is maintained or even surpassed by the implicit batch size scheduling of RASDA. With RASDA, systematic HPO is applied to a terabyte-scale scientific dataset for the first time in the literature, enabling efficient optimization of complex models on massive scientific data.

The trend of building larger and more complex imaging satellite constellations leads to the challenge in managing multiple acquisition requests of the Earth surface. Optimally planning these acquisitions is an intractable optimization problem, and heuristic algorithms are used today for finding sub-optimal solutions. Recently, quantum algorithms have been considered for this purpose, due to the potential breakthroughs that they can bring in optimization, expecting either a speedup or an increase in the solution quality. Hybrid quantum-classical methods have been considered as a short-term solution for taking advantage of small quantum machines. In this paper, we propose reverse quantum annealing as a method for improving the acquisition plan obtained by a classical optimizer. We investigate the benefits of the method with different annealing schedules and different problem sizes. The obtained results provide guidelines on designing a larger hybrid quantum-classical framework based on reverse quantum annealing for this application.

The increasing availability of high-resolution, open-access satellite data facilitates the production of global Land Cover (LC) maps, an essential source of information for managing and monitoring natural and human-induced processes. However, the accuracy of the obtained LC maps can be affected by the discrepancy between the spatial resolution of the satellite images and the extent of the LC present in the scene. Indeed, several pixels may be misclassified because of their mixed spectral signatures, i.e., more than two LC classes are present in the pixel. To solve this problem, this paper proposes an approach that explores the possibility of using simple but effective unmixing approaches to enhance the classification accuracy of the mixed spectral pixels. The results showed that several pixels, including buildings and grassland LC, are typically classified as cropland. By unmixing their spectral content, it is possible to extract the most prevalent class within the area of each pixel to update the classification map, thus sharply increasing the map accuracy. These promising preliminary results indicate the potential for broader applicability and efficiency in global LC mapping.

Seismic Imaging (SI) survey design for onshore applications faces challenges such as accessibility and poor data quality due to unexpected (near-)surface conditions. In this paper, we explore the correlation between the surface conditions provided by Land-Cover (LC) maps generated using Remote Sensing (RS) data and different settings of Seismic Processing (SP) parameters. The study involves a 2D seismic line related to geothermal exploration from the Netherlands.

VRoot is an immersive extended reality reconstruction tool for root system architectures from 3D volumetric scans of soil columns. We have conducted a laboratory user study to assess the performance of new users with our software in comparison to established software. We utilize a plant model to derive a synthetic root architecture, providing a baseline for reconstruction. This demo showcases the processes and techniques contributing to exact and efficient manual root architecture reconstruction in Virtual Reality. The extraction task typically is the sparse graph-structure extraction from a 3D magnetic-resonance imaging (MRI) data set. We visualize the RSA directly within the MRI and offer selection-set-based methods of adapting and augmenting the root architecture. This application is in productive use at our partner institute, where it is used to analyze complex root images.

The main objective of the ADMIRE project1 is the creation of an active I/O stack that dynamically adjusts computation and storage requirements through intelligent global coordination, the elasticity of computation and I/O, and the scheduling of storage resources along all levels of the storage hierarchy, while offering quality-of-service (QoS), energy efficiency, and resilience for accessing extremely large data sets in very heterogeneous computing and storage environments. We have developed a framework prototype that is able to dynamically adjust computation and storage requirements through intelligent global coordination, separated control, and data paths, the malleability of computation and I/O, the scheduling of storage resources along all levels of the storage hierarchy, and scalable monitoring techniques. The leading idea in ADMIRE is to co-design applications with ad-hoc storage systems that can be deployed with the application and adapt their computing and I/O behaviour on runtime, using malleability techniques, to increase the performance of applications and the throughput of the applications.

Kernel methods are class of Machine Learning (ML) models that have been widely employed in the literature for Earth Observation (EO) applications. The increasing development of quantum computing hardware motivates further research to improve the capabilities and the performances of data analysis algorithms. In this manuscript an implementation of Adiabatic Quantum Kitchen Sinks (AQKS) kernel estimation algorithm integrated with parallel quantum annealing is presented. Such combination with the concept of parallel quantum annealing allows for the solving of multiple problem instances in the same annealing cycle, thus reducing the number of rquired calls to the quantum annealing solver. The proposed workflow is then implemented using a D-Wave Advantage system and tested on a regression problem on a real Remote Sensing (RS) dataset. The obtained results are then analyzed and compared with those obtained by a classical kernel approximation algorithm based on Random Fourier Features.

The continuous stream of high spatial resolution satellite data offers the opportunity to regularly produce land cover (LC) maps. To this end, Transformer deep learning (DL) models have recently proven their effectiveness in accurately classifying long time series (TS) of satellite images. The continual generation of regularly updated LC maps can be used to analyze dynamic phenomena and extract multi-temporal information. However, several challenges need to be addressed. Our paper aims to study how the performance of a Transformer model changes when classifying TS of satellite images acquired in years later than those in the training set. In particular, the behavior of the attention in the Transformer model is analyzed to determine when the information provided by the initial training set needs to be updated to keep generating accurate LC products. Preliminary results show that: (i) the selection of the positional encoding strategy used in the Transformer has a significant impact on the classification accuracy obtained with multi-year TS, and (ii) the most affected classes are the seasonal ones.

Earth Observation (EO) data processing faces challenges due to large volumes, multiple sources, and diverse formats. To address this issue, this paper presents a scalable and parallelizable workflow using Apache Airflow, capable of integrating Machine Learning (ML) and Deep Learning (DL) models with Modular Supercomputing Architecture (MSA) systems. To test the workflow, we considered the production of large-scale Land-Cover (LC) maps as a case study. The workflow manager, Airflow, offers scalability, extensibility, and programmable task definition in Python. It allows us to execute different steps of the workflow in different High-Performance Computing (HPC) systems. The workflow is demonstrated on the Dynamical Exascale Entry Platform (DEEP) and Jülich Research on Exascale Cluster Architectures (JURECA) hosted at the Jülich Supercomputing Centre (JSC), a platform that incorporates heterogeneous JSC systems.

Land cover maps generated by the classification of remote sensing data allow for monitoring Earth processes and the dynamics of objects and phenomena. For accurate land cover variability quantification in environmental monitoring, maps need to be spatiotemporally consistent, continually updated, and indicate permanent changes. However, producing frequent and spatiotemporally consistent land cover maps is challenging because it involves balancing the need for temporal consistency with the risk of missing real changes. In this work, we propose a scalable and semi-automatic method for generating annual land cover maps with labels that are consistently applied from one year to the next. It uses a Transformer deep learning model as a classifier, which is trained on satellite time series of images using High Performance Computing (HPC). The trained model can generate stable maps by shifting the prediction window along the temporal direction. The effectiveness of the proposed approach is tested qualitatively and quantitatively on a multi-annual Sentinel-2 dataset acquired over a three-year period in a study area located in the southern Italian Alps.

The design, development, and deployment of scientific computing applications can be quite complex, in particular when involving Machine Learning (ML) or High-Performance Computing (HPC). They require scientific and software engineering expertise and in addition HPC or ML knowledge. Often, such applications are however developed by scientists who are experts in their domain, but need support for the software engineering, ML, and HPC aspects. The cooperation and communication between experts from these quite different disciplines can be difficult though. We therefore propose to employ the Interaction Room (IR), a method that facilitates interdisciplinary collaboration in complex software projects. An IR uses annotated drawings to exchange information and stimulate discussion between project stakeholders, in order to improve common understanding and identify uncertainties, risks, and other aspects that are critical to a project's success early on. We suggest different drawing canvases and annotations that focus on different viewpoints and issues of the project. These canvases are specific to the project type, such as ML applications or HPC simulations.

Deep Learning models have proven necessary in dealing with the challenges posed by the continuous growth of data volume acquired from satellites and the increasing complexity of new Remote Sensing applications. To obtain the best performance from such models, it is necessary to fine-tune their hyperparameters. Since the models might have massive amounts of parameters that need to be tuned, this process requires many computational resources. In this work, a method to accelerate hyperparameter optimization on a High-Performance Computing system is proposed. The data batch size is increased during the training, leading to a more efficient execution on Graphics Processing Units. The experimental results confirm that this method reduces the runtime of the hyperparameter optimization step by a factor of 3 while achieving the same validation accuracy as a standard training procedure with a fixed batch size.

The implementation of scalable processing workflows is essential to improve the access to and analysis of the vast amount of high-resolution and multi-source Remote Sensing (RS) data and to provide decision-makers with timely and valuable information. The Modular Supercomputing Architecture (MSA) systems that are operated by the Jülich Supercomputing Centre (JSC) are a concrete solution for data-intensive RS applications that rely on big data storage and processing capabilities. To meet the requirements of applications with more complex computational tasks, JSC plans to connect the High Performance Computing (HPC) systems of its MSA environment to different quantum computers via the Jülich UNified Infrastructure for Quantum computing (JUNIQ). The paper describes this unique computing environment and highlights its potential to address real RS application scenarios through high-performance and hybrid quantum-classical processing workflows.

Regression analysis has a crucial role in many Earth Ob-servation (EO) applications. The increasing availability and recent development of new computing technologies moti-vate further research to expand the capabilities and enhance the performance of data analysis algorithms. In this paper, the biophysical variable estimation problem is addressed. A novel approach is proposed, which consists in a reformulated Support Vector Regression (SVR) and leverages Quantum Annealing (QA). In particular, the SVR optimization prob-lem is reframed to a Quadratic Unconstrained Binary Opti-mization (QUBO) problem. The algorithm is then tested on the D-Wave Advantage quantum annealer. The experiments presented in this paper show good results, despite current hardware limitations, suggesting that this approach is viable and has great potential.

This paper presents an approach that aims to produce a Time-Series (TS) of consistent Land-Cover (LC) maps, typically needed to perform environmental monitoring. First, it creates an annual training set for each TS to be classified, leveraging on publicly available thematic products. These annual training sets are then used to generate a set of preliminary LC maps that allow for the identification of the unchanged areas, i.e., the stable temporal component. Such areas can be used to define an informative and reliable multi-year training set, by selecting samples belonging to the different years for all the classes. The multi-year training set is finally employed to train a unique multi-year Long Short Term Mem-ory (LSTM) model, which enhances the consistency of the annual LC maps. The preliminary results carried out on three TSs of Sentinel 2 images acquired in Italy in 2018,2019 and 2020 demonstrates the capability of the method to improve the consistency of the annual LC maps. The agreement of the obtained maps is$\approx 78{\%}$, compared to the$\approx 74{\%}$achieved by the LSTM models trained separately.

In Remote Sensing (RS) classification, generalization ability is one of the measure that characterizes the success of Machine Learning (ML) models, but is often impeded by the scarse availability of annotated training data. Annotated RS samples are expensive to obtain and can present large disparities when produced by different annotators. In this paper, we utilize Few-Shot Learning (FSL) with meta-learning to address the challenge of generalization using limited amount of training information. The data used in this paper is leveraged from different datasets that have diverse distributions, that means distinct feature spaces. We tested our approach on publicly available RS benchmark datasets to perform few-shot RS image classification using meta-learning. The results of the experiments suggest that our approach is able to generalize well on the unseen data even with limited number of training samples and reasonable training time.

The increasing availability of quantum computers motivates researching their potential capabilities in enhancing the performance of data analysis algorithms. Similarly, as in other research communities, also in Remote Sensing (RS) it is not yet defined how its applications can benefit from the usage of quantum computing. This paper proposes a formulation of the Support Vector Regression (SVR) algorithm that can be executed by D-Wave quantum computers. Specifically, the SVR is mapped to a Quadratic Unconstrained Binary Optimization (QUBO) problem that is solved with Quantum Annealing (QA). The algorithm is tested on two different types of computing environments offered by D-Wave: The Advantage system, which directly embeds the problem into the Quantum Processing Unit (QPU), and a Hybrid solver that employs both classical and quantum computing resources. For the evaluation, we considered a biophysical variable estimation problem with RS data. The experimental results show that the proposed quantum SVR implementation can achieve comparable or in some cases better results than the classical implementation. This work is one of the first attempts to provide insight into how QA could be exploited and integrated in future RS workflows based on Machine Learning (ML) algorithms.

BigEarthNet is one of the standard large remote sensing datasets. It has been shown previously that neural networks are effective tools to classify the image patches in this data. However, finding the optimum network hyperparameters and architecture to accurately classify the image patches in BigEarthNet remains a challenge. Searching for more accurate models manually is extremely time consuming and labour intensive. Hence, a systematic approach is advisable. One possibility is automated evolutionary Neural Architecture Search (NAS). With this NAS many of the commonly used network hyperparameters, such as loss functions, are eliminated and a more accurate network is determined.

Recent developments in Quantum Computing (QC) have paved the way for an enhancement of computing capabilities. Quantum Machine Learning (QML) aims at developing Machine Learning (ML) models specifically designed for quantum computers. The availability of the first quantum processors enabled further research, in particular the exploration of possible practical applications of QML algorithms. In this work, quantum formulations of the Support Vector Machine (SVM) are presented. Then, their implementation using existing quantum technologies is discussed and Remote Sensing (RS) image classification is considered for evaluation.

Using computationally efficient techniques for transforming the massive amount of Remote Sensing (RS) data into scientific understanding is critical for Earth science. The utilization of efficient techniques through innovative computing systems in RS applications has become more widespread in recent years. The continuously increased use of Deep Learning (DL) as a specific type of Machine Learning (ML) for data-intensive problems (i.e., ‘big data’) requires powerful computing resources with equally increasing performance. This paper reviews recent advances in High-Performance Computing (HPC), Cloud Computing (CC), and Quantum Computing (QC) applied to RS problems. It thus represents a snapshot of the state-of-the-art in ML in the context of the most recent developments in those computing areas, including our lessons learned over the last years. Our paper also includes some recent challenges and good experiences by using Europeans fastest supercomputer for hyper-spectral and multi-spectral image analysis with state-of-the-art data analysis tools. It offers a thoughtful perspective of the potential and emerging challenges of applying innovative computing paradigms to RS problems.

A wide variety of Remote Sensing (RS) missions are continuously acquiring a large volume of data every day. The availability of large datasets has propelled Deep Learning (DL) methods also in the RS domain. Convolutional Neural Networks (CNNs) have become the state of the art when tackling the classification of images, however the process of training is time consuming. In this work we exploit the Layer-wise Adaptive Moments optimizer for Batch training (LAMB) optimizer to use large batch size training on High-Performance Computing (HPC) systems. With the use of LAMB combined with learning rate scheduling and warm-up strategies, the experimental results on RS data classification demonstrate that a ResNet50 can be trained faster with batch sizes up to 32K.

Support Vector Machine (SVM) is a popular supervised Machine Learning (ML) method that is widely used for classification and regression problems. Recently, a method to train SVMs on a D-Wave 2000Q Quantum Annealer (QA) was proposed for binary classification of some biological data. First, ensembles of weak quantum SVMs are generated by training each classifier on a disjoint training subset that can be fit into the QA. Then, the computed weak solutions are fused for making predictions on unseen data. In this work, the classification of Remote Sensing (RS) multispectral images with SVMs trained on a QA is discussed. Furthermore, an open code repository is released to facilitate an early entry into the practical application of QA, a new disruptive compute technology.

Similarly to other scientific domains, Deep Learning (DL) holds great promises to fulfil the challenging needs of Remote Sensing (RS) applications. However, the increase in volume, variety and complexity of acquisitions that are carried out on a daily basis by Earth Observation (EO) missions generates new processing and storage challenges within operational processing pipelines. The aim of this work is to show that High-Performance Computing (HPC) systems can speed up the training time of Convolutional Neural Networks (CNNs). Particular attention is put on the monitoring of the classification accuracy that usually degrades when using large batch sizes. The experimental results of this work show that the training of the model scales up to a batch size of 8,000, obtaining classification performances in terms of accuracy in line with those using smaller batch sizes.

The classification of land-cover classes in remote sensing images can suit a variety of interdisciplinary applications such as the interpretation of natural and man-made processes on the Earth surface. The Convolutional Support Vector Machine (CSVM) network was recently proposed as binary classifier for the detection of objects in Unmanned Aerial Vehicle (UAV) images. The training phase of the CSVM is based on convolutional layers that learn the kernel weights via a set of linear Support Vector Machines (SVMs). This paper proposes the Multi-scale Convolutional Support Vector Machine (MCSVM) network, that is an ensemble of CSVM classifiers which process patches of different spatial sizes and can deal with multi-class classification problems. The experiments are carried out on the EuroSAT Sentinel-2 dataset and the results are compared to the one obtained with recent transfer learning approaches based on pre-trained Convolutional Neural Networks (CNNs).

The implementation of efficient remote sensing workflows is essential to improve the access to and analysis of the vast amount of sensed data and to provide decision-makers with clear, timely, and useful information. The Dynamical Exascale Entry Platform (DEEP) is an European pre-exascale platform that incorporates heterogeneous High-Performance Computing (HPC) systems, i.e., hardware modules which include specialised accelerators. This paper demonstrates the potential of such diverse modules for the deployment of remote sensing data workflows that include diverse processing tasks. Particular focus is put on pipelines which can use the Network Attached Memory (NAM), which is a novel supercomputer module that allows near processing and/or fast shared storage of big remote sensing datasets.

Advances in remote sensing hardware have led to a significantly increased capability for high-quality data acquisition, which allows the collection of remotely sensed images with very high spatial, spectral, and radiometric resolution. This trend calls for the development of new techniques to enhance the way that such unprecedented volumes of data are stored, processed, and analyzed. An important approach to deal with massive volumes of information is data compression, related to how data are compressed before their storage or transmission. For instance, hyperspectral images (HSIs) are characterized by hundreds of spectral bands. In this sense, high-performance computing (HPC) and high-throughput computing (HTC) offer interesting alternatives. Particularly, distributed solutions based on cloud computing can manage and store huge amounts of data in fault-tolerant environments, by interconnecting distributed computing nodes so that no specialized hardware is needed. This strategy greatly reduces the processing costs, making the processing of high volumes of remotely sensed data a natural and even cheap solution. In this paper, we present a new cloud-based technique for spectral analysis and compression of HSIs. Specifically, we develop a cloud implementation of a popular deep neural network for non-linear data compression, known as autoencoder (AE). Apache Spark serves as the backbone of our cloud computing environment by connecting the available processing nodes using a master-slave architecture. Our newly developed approach has been tested using two widely available HSI data sets. Experimental results indicate that cloud computing architectures offer an adequate solution for managing big remotely sensed data sets.

Progress in sensor technology leads to an ever-increasing amount of remote sensing data which needs to be classified in order to extract information. This big amount of data requires parallel processing by running parallel implementations of classification algorithms, such as Support Vector Machines (SVMs), on High-Performance Computing (HPC) clusters. Tomorrow's supercomputers will be able to provide exascale computing performance by using specialised hardware accelerators. However, existing software processing chains need to be adapted to make use of the best fitting accelerators. To address this problem, a mapping of an SVM remote sensing classification chain to the Dynamical Exascale Entry Platform (DEEP), a European pre-exascale platform, is presented. It will allow to scale SVM-based classifications on tomorrow's hardware towards exascale performance.

Supervised image classification is one of the essential techniques for generating semantic maps from remotely sensed images. The lack of labeled ground truth datasets, due to the inherent time effort and cost involved in collecting training samples, has led to the practice of training and validating new classifiers within a single image. In line with that, the dominant approach for the division of the available ground truth into disjoint training and test sets is random sampling. This paper discusses the problems that arise when this strategy is adopted in conjunction with spectral-spatial and pixel-wise classifiers such as 3D Convolutional Neural Networks (3D CNN). It is shown that a random sampling scheme leads to a violation of the independence assumption and to the illusion that global knowledge is extracted from the training set. To tackle this issue, two improved sampling strategies based on the Density-Based Clustering Algorithm (DBSCAN) are proposed. They minimize the violation of the train and test samples independence assumption and thus ensure an honest estimation of the generalization capabilities of the classifier.

The progress of remote sensing technologies leads to increased supply of high-resolution image data. However, solutions for processing large volumes of data are lagging behind: desktop computers cannot cope anymore with the requirements of macro-scale remote sensing applications; therefore, parallel methods running in High-Performance Computing (HPC) environments are essential. Managing an HPC processing pipeline is non-trivial for a scientist, especially when the computing environment is heterogeneous and the set of tasks has complex dependencies. This paper proposes an end-to-end scientific workflow approach based on the UNICORE workflow management system for automating the full chain of Support Vector Machine (SVM)-based classification of remotely sensed images. The high-level nature of UNICORE workflows allows to deal with heterogeneity of HPC computing environments and offers powerful workflow operations such as needed for parameter sweeps. As a result, the remote sensing workflow of SVM-based classification becomes re-usable across different computing environments, thus increasing usability and reducing efforts for a scientist.

Component trees are region-based representations that encode the inclusion relationship of the threshold sets of an image. These representations are one of the most promising strategies for the analysis and the interpretation of spatial information of complex scenes as they allow the simple and efficient implementation of connected filters. This work proposes a new efficient hybrid algorithm for the parallel computation of two particular component trees-the max- and min-tree-in shared and distributed memory environments. For the node-local computation a modified version of the flooding-based algorithm of Salembier is employed. A novel tuple-based merging scheme allows to merge the acquired partial images into a globally correct view. Using the proposed approach a speed-up of up to 44.88 using 128 processing cores on eight-bit gray-scale images could be achieved. This is more than a five-fold increase over the state-of-the-art shared-memory algorithm, while also requiring only one-thirty-second of the memory.

Self-Dual Attribute Profiles (SDAPs) have proven to be an effective method for extracting spatial features able to improve scene classification of remote sensing images with very high spatial resolution. An SDAP is a multilevel decomposition of an image obtained with a sequence of transformations performed by attribute filters over the Tree of Shapes (ToS). One of the main issues with this technique is the identification of the filter thresholds generating a SDAP composed of features that should be relevant for the classification problem. This paper proposes a tree-based supervised feature extraction strategy, which is based on Fisher's linear discriminant analysis relying on the available class information. The exploitation of the ToS structure in the threshold selection procedure allows one to avoid any prior full image filtering, as in other related techniques. Furthermore, the ToS automates and optimizes the whole process by decreasing the computational time and overcoming the conventional selection procedure based on trial and error attempts. The proposed automatic spatial feature extraction technique has been tested in the classification of a very high resolution image proving its effectiveness with respect to a conventional selection strategy.

Classification of remote sensing images often use Support Vector Machines (SVMs) that require an n-fold cross-validation phase in order to do model selection. This phase is characterized by sweeping through a wide set of parameter combinations of SVM kernel and cost parameters. As a consequence this process is computationally expensive but represents a principled way of tuning a model for better accuracy and to prevent overfitting together with regularization that is in SVMs inherently solved in the optimization. Since the cross-validation technique is done in a principled way also known as `gridsearch', we aim at supporting remote sensing scientists in two ways. Firstly by reducing the time-to-solution of the cross-validation by applying state-of-the-art parallel processing methods because the sweep of parameters and cross-validation runs itself can be nicely parallelized. Secondly by reducing manual labour by automating the parallel submission processes since manually performing cross-validation is very time consuming, unintuitive, and error-prone especially in large-scale cluster or supercomputing environments (e.g., batch job scripts, node/core/task parameters, etc.).

In this paper, we propose an instrumentation and computer vision pipeline that allows automatic object detection on images taken from multiple experimental set ups. We demonstrate the approach by autonomously counting intoxicated flies in the FLORIDA assay. The assay measures the effect of ethanol exposure onto the ability of a vinegar fly Drosophila melanogaster to right itself. The analysis consists of a three-step approach. First, obtaining an image of a large set of individual experiments, second, identify areas containing a single experiment, and third, discover the searched objects within the experiment. For the analysis we facilitate well-known computer vision and machine learning algorithms - namely color segmentation, threshold imaging and DBSCAN. The automation of the experiment enables an unprecedented reproducibility and consistency, while significantly decreasing the manual labor.

Big Data Analytics methods take advantage of techniques from the fields of data mining, machine learning, or statistics with a focus on analysing large quantities of data (aka `big datasets') with modern technologies. Big data sets appear in remote sensing in the sense of large volumes, but also in the sense of an ever increasing amount of spectral bands (i.e., high-dimensional data). The remote sensing has traditionally used the above described techniques for a wide variety of application such as classification (e.g., land cover analysis using different spectral bands from satellite data), but more recently scalability challenges occur when using traditional (often serial) methods. This paper addresses observed scalability limits when using support vector machines (SVMs) for classification and discusses scalable and parallel developments used in concrete application areas of remote sensing. Different approaches that are based on massively parallel methods are discussed as well as recent developments in parallel methods.

The big data analytics approach emerged that can be interpreted as extracting information from large quantities of scientific data in a systematic way. In order to have a more concrete understanding of this term we refer to its refinement as smart data analytics in order to examine large quantities of scientific data to uncover hidden patterns, unknown correlations, or to extract information in cases where there is no exact formula (e.g. known physical laws). Our concrete big data problem is the classification of classes of land cover types in image-based datasets that have been created using remote sensing technologies, because the resolution can be high (i.e. large volumes) and there are various types such as panchromatic or different used bands like red, green, blue, and nearly infrared (i.e. large variety). We investigate various smart data analytics methods that take advantage of machine learning algorithms (i.e. support vector machines) and state-of-the-art parallelization approaches in order to overcome limitations of big data processing using non-scalable serial approaches.

SUMMARY The UltraScan data analysis application is a software package that is able to take advantage of computational resources in order to support the interpretation of analytical ultracentrifugation experiments. Since 2006, the UltraScan scientific gateway has been used with Web browsers in TeraGrid by scientists studying the solution properties of biological and synthetic molecules. UltraScan supports its users with a scientific gateway in order to leverage the power of supercomputing. In this contribution, we will focus on several advancements of the UltraScan scientific gateway architecture with a standardized job management while retaining its lightweight design and end user interaction experience. This paper also presents insights into a production deployment of UltraScan in Europe. The approach is based on open standards with respect to job management and submissions to the Extreme Science and Engineering Discovery Environment in the USA and to similar infrastructures in Europe such as the European Grid Infrastructure or the Partnership for Advanced Computing in Europe (PRACE). Our implementation takes advantage of the Apache Airavata framework for scientific gateways that lays the foundation for easy integration into several other scientific gateways. Copyright © 2014 John Wiley & Sons, Ltd.

Parameter sweeps are used by researchers with scientific domain-specific tools or workflows to submit a large collection of computational jobs whereby each single job of it only varies in certain parts. They require a more fine-grained distribution of jobs across resources, which also raise a significant challenge for efficient resource management in middleware environments that have been not specifically designed to perform parameter sweeps. This paper offers insights into parameter sweep solutions that support multi-disciplinary science environments via abstraction from resource management complexities using middleware. The solutions are based on use case requirements, enable efficient submission, enhanced usability, and standard compliance. We also apply a use case taken from the life science domain to demonstrate usefulness and efficiency of the solutions.

The last two decades have seen an exceptional increase of the available networking, computing and storage resources. Scientific research communities have exploited these enhanced capabilities developing large scale collaborations, supported by distributed infrastructures. In order to enable usage of such infrastructures, several middleware solutions have been created. However such solutions, having been developed separately, have been resulting often in incompatible middleware and infrastructures. The European Middleware Initiative (EMI) is a collaboration, started in 2010, among the major European middleware providers (ARC, dCache, gLite, UNICORE), aiming to consolidate and evolve the existing middleware stacks, facilitating their interoperability and their deployment on large distributed infrastructures, establishing at the same time a sustainable model for the future maintenance and evolution of the middleware components. This paper presents the strategy followed for the achievements of these goals : after an analysis of the situation before EMI, it is given an overview of the development strategy, followed by the most notable technical results, grouped according to the four development areas (Compute, Data, Infrastructure, Security). The rigorous process ensuring the quality of provided software is then illustrated, followed by a description the release process, and of the relations with the user communities. The last section provides an outlook to the future, focusing on the undergoing actions looking toward the sustainability of activities.

In September 2011 the European Middleware Initiative (EMI) started discussing the feasibility of creating an open source community for science with other projects like EGI, StratusLab, OpenAIRE, iMarine, and IGE, SMEs like DCore, Maat, SixSq, SharedObjects, communities like WLCG and LSGC. The general idea of establishing an open source community dedicated to software for scientific applications was understood and appreciated by most people. However, the lack of a precise definition of goals and scope is a limiting factor that has also made many people sceptical of the initiative. In order to understand more precisely what such an open source initiative should do and how, EMI has started a more formal feasibility study around a concept called ScienceSoft - Open Software for Open Science. A group of people from interested parties was created in December 2011 to be the ScienceSoft Steering Committee with the short-term mandate to formalize the discussions about the initiative and produce a document with an initial high-level description of the motivations, issues and possible solutions and a general plan to make it happen. The conclusions of the initial investigation were presented at CERN in February 2012 at a ScienceSoft Workshop organized by EMI. Since then, presentations of ScienceSoft have been made in various occasions, in Amsterdam in January 2012 at the EGI Workshop on Sustainability, in Taipei in February at the ISGC 2012 conference, in Munich in March at the EGI/EMI Conference and at OGF 34 in March. This paper provides information this concept study ScienceSoft as an overview distributed to the broader scientific community to critique it.

E-health makes use of information and communication methods and the latest e-research tools to support the understanding of body functions. E-scientists in this field take already advantage of one single infrastructure to perform computationally-intensive investigations of the human body that tend to consider each of the constituent parts separately without taking into account the multiple important interactions between them. But these important interactions imply an increasing complexity of applications that embrace multiple physical models (i.e. multi-physics) and consider a larger range of scales (i.e. multi-scale) thus creating a steadily growing demand for interoperable infrastructures that allow for new innovative application types of jointly using different infrastructures for one application. But interoperable infrastructures are still not seamlessly provided and we argue that this is due to the absence of a realistically implementable infrastructure interoperability reference model that is based on lessons learned from e-science usage. Therefore, the goal of this paper is to explore the potential of using multiple infrastructures for one scientific goal with a particular focus on e-health. Since e-scientists gain more interest in using multiple infrastructures there is a clear demand for interoperability between them to enable a use with one e-research tool. The paper highlights work in the context of an e-Health blood flow application while the reference model is applicable to other e-science applications as well.

In the last decade, life science applications have become more and more integrated into e-Science environments, hence they are typically very demanding, both in terms of computational capabilities and data capacities. Especially the access to life science applications, embedded in such environments via Grid clients still constitutes a major hurdle for scientists that do not have an IT background. Life science applications often comprise a whole set of small programs instead of a single executable. Many of the graphical Grid clients are not perfectly suited for these types of applications, as they often assume that Grid jobs will run a single executable instead of a set of chained executions (i.e. sequences). This means that in order to execute a sequence of multiple programs on a single Grid resource, piping data from one program to the next, the user would have to run a hand-written shell script. Otherwise each program is independently scheduled as a Grid job, which causes unnecessary file transfers between the jobs, even if they are scheduled on the same resource. We present a generic solution to this problem and provide a reference implementation, which seamlessly integrates with the Grid middleware UNICORE. Our approach focuses on a comfortable user interface for the creation of such program sequences, validated in UNICORE-driven HPC-based Grids. Thus, we applied our approach in order to provide support for the usage of the AMBER package (a widely-used collection of programs for molecular dynamics simulations) within Grid workflows. We finally provide a scientific use case of our approach leveraging the interoperability of two different scientific infrastructures that represents an instance of the infrastructure interoperability reference model.

Abstract Many production Grid and e‐Science infrastructures have begun to offer services to end‐users during the past several years with an increasing number of scientific applications that require access to a wide variety of resources and services in multiple Grids. Therefore, the Grid Interoperation Now—Community Group of the Open Grid Forum—organizes and manages interoperation efforts among those production Grid infrastructures to reach the goal of a world‐wide Grid vision on a technical level in the near future. This contribution highlights fundamental approaches of the group and discusses open standards in the context of production e‐Science infrastructures. Copyright © 2009 John Wiley & Sons, Ltd.

Simulation and thus scientific computing is the third pillar alongside theory and experiment in todays science and engineering. The term e-science evolved as a new research field that focuses on collaboration in key areas of science using next generation infrastructures to extend the powers of scientific computing. This paper contributes to the field of e-science as a study of how scientists actually work within currently existing Grid and e-science infrastructures. Alongside numerous different scientific applications, we identified several common approaches with similar characteristics in different domains. These approaches are described together with a classification on how to perform e-science in next generation infrastructures. The paper is thus a survey paper which provides an overview of the e-science research domain.

In a distributed grid environment with ambitious service demands the job submission and management interfaces provide functionality of major importance. Emerging e-science and grid infrastructures such as EGEE and DEISA rely on highly available services that are capable of managing scientific jobs. It is the adoption of emerging open standard interfaces which allows the distribution of grid resources in such a way that their actual service implementation or grid technologies are not isolated from each other, especially when these resources are deployed in different e-science infrastructures that consist of different types of computational resources. This paper motivates the interoperability of these infrastructures and discusses solutions. We describe the adoption of various open standards that recently emerged from the open grid forum (OGF) in the field of job submission and management by well-known grid technologies, respectively gLite and UNICORE. This has a fundamental impact on the interoperability between these technologies and thus within the next generation e-science infrastructures that rely on these technologies.

In the past several years, many scientific applications from various domains have taken advantage of e-science infrastructures that share storage or computational resources such as supercomputers, clusters or PC server farms across multiple organizations. Especially within e-science infrastructures driven by high-performance computing (HPC) such as DEISA, online visualization and computational steering (COVS) has become an important technique to save compute time on shared resources by dynamically steering the parameters of a parallel simulation. This paper argues that future supercomputers in the Petaflop/s performance range with up to 1 million CPUs will create an even stronger demand for seamless computational steering technologies. We discuss upcoming challenges for the development of scalable HPC applications and limits of future storage/IO technologies in the context of next generation e- science infrastructures and outline potential solutions.

Many existing Grid technologies and resource management systems lack a standardized job submission interface in Grid environments or e-Infrastructures. Even if the same language for job description is used, often the interface for job submission is also different in each of these technologies. The evolvement of the standardized Job Submission and Description Language (JSDL) as well as the OGSA - Basic Execution Services (OGSA-BES) pave the way to improve the interoperability of all these technologies enabling cross-Grid job submission and better resource management capabilities. In addition, the BytelO standards provide useful mechanisms for data access that can be used in conjunction with these improved resource management capabilities. This paper describes the integration of these standards into the recently released UNICORE 6 Grid middleware that is based on open standards such as the Web Services Resource Framework (WS-RF) and WS-Addressing (WS-A).

Many production Grid infrastructures such as DEISA, EGEE, or TeraGrid have begun to offer services to endusers that include access to computational resources. The major goal of these infrastructures is to facilitate the routine interaction of scientists and their workflows with advanced tools and seamless access to computational resources via Grid middleware systems such as UNICORE, gLite or Globus Toolkits. While UNICORE 5 is used in production Grids since several years, recently an early prototype of the new Web services-based UNICORE 6 became available that will be continuously improved in the next months for its use in production. In absence of a widely accepted framework for visualization and steering, the new UNICORE 6 Grid middleware provides not such a higher level service by default. This motivates this contribution to support e-Scientists in upcoming WS-based UNICORE Grids with visualization and steering techniques. In this paper we present the augmentation of the early standards-based UNICORE 6 prototype with a higher-level service for collaborative online visualization and steering. It describes the seamless integration of this service within UNICORE Grids by retaining the convenient single sign-on feature.

Large-scale scientific research often relies on the collaborative use of Grid and e-Science infrastructures that provide computational or storage related resources. One of the ideas of these modern infrastructures is to facilitate the routine interaction of scientists and their workflows with advanced problem solving tools and computational resources. While many production Grid projects and e-Science infrastructures have begun to offer services for the usage of resources to end-users during the past several years, the corresponding emerging standards defined by GGF and OASIS still appear to be in flux. In this paper, we present the GridBean technology that bridges the gap between the constantly changing basic Grid or e-Science infrastructures and the need of stable application development environments for the Grid users.

The UNICORE grid technology provides a seamless, secure and intuitive access to distributed grid resources such as computational or storage related resources. In addition, its extensible character through application-specific plug-ins and its enhancements developed in various European-funded projects leads to the UNICORE technology that is used in daily production at many supercomputer centers and research facilities world-wide today. In this paper we present an enhancement that provides the dynamic capabilities of a secure shell terminal within the UNICORE grid technology while single sign-on remains. This enhancement allows the integration of the dynamic work-behavior of scientists, or existing scientific applications, to be more integrated into the usual workflow with UNICORE and therefore in collaborative grid environments. As a well-known tool in the scientific community, a secure shell terminal provides the most flexible way of working on remote systems that no graphical user interface or advanced tooling in grid computing can ever provide