Although Machine Learning (ML) is widely used in a variety of interdisciplinary applications, its implementation in safety-critical systems, such as the Attitude Control System (ACS) of a satellite, poses numerous challenges. While previous studies have shown promising results, there is a lack of information on the design and development process for the application of ML in real-time control systems. This paper presents the implementation of a Deep Reinforcement Learning (DRL) model for a magnetic-based ACS of the UPMSat-2 satellite. The primary objective is not only to design, implement, and validate an RL agent, but also to provide some insights and criteria of the decision-making process to achieve an adequate model. The system was trained and validated on a simulation model with positive results. To further validate non-functional requirements, the resulting trained agent was tested on a real-time embedded system according to safety standards. The obtained quantitative metrics and performance results show the ability of the agent to maintain the satellite’s attitude across various operational phases, leveraging its adaptability to dynamic conditions.

Virtual knowledge graphs (VKGs) have been widely applied to access relational data with a semantic layer by using an ontology in use cases that are dynamic in nature. However, current VKG techniques focus mainly on accessing a single relational database and remain largely unstudied for data integration with several heterogeneous data sources. To overcome this limitation, we propose intermediate triple table ( ITT ), a general VKG architecture to access multiple and diverse data sources. Our proposal is based on data shipping and addresses heterogeneity by adopting a schema-oblivious graph representation that intervenes between the sources and the queries. We minimize data computation by just materializing a relevant subgraph for a specific query. We employ star-shaped query processing and extend this technique to mapping candidate selection. For rapid materialization of the ITT , we apply a mapping partitioning technique to parallelize mapping execution, which also guarantees duplicate-free subgraphs and reduces memory consumption. We use SPARQL-to-SQL query translation to homogeneously evaluate queries over the ITT and execute them with an in-process analytical store. We implemented ITT on top of a knowledge graph materialization engine and evaluated it with two VKG benchmarks. The experimental results show that our proposal outperforms state-of-the-art techniques for complex graph queries in terms of execution time. It also decreases the number of timeouts although it uses more memory as a trade-off. The experiments also demonstrate the source independence of the architecture on a mixed distribution of data with SQL and document stores together with various file formats.

Recently, air quality has become a major concern for the protection of the environment and the well-being of people. Air pollution is a key proxy of the quality of life in any city and is crucial in the fight against climate change. Therefore, it is of utmost importance to analyze and forecast the concentration of pollutants. Current efforts in these tasks tend to focus on individual gases and rely on a static view of the measuring sensors. This work presents an Artificial Intelligence framework that forecasts the concentrations of eleven pollutants in a Region Of Interest for several forecast horizons. The framework is based on Convolutional Long Short-Term Memory networks. Unlike the reviewed state-of-the-art, this framework can work with multiple pollutants simultaneously. Furthermore, it can incorporate exogenous input data by means of independent modules. The framework can also deal with newly installed sensors and recover from missing data. An extensive 10-year hourly dataset was used to train and validate the framework. The aim of this work is to provide an operational air quality framework for real-world application. This implies not only modeling future air quality but also supporting sensor failures and adaptation to different pollutants. None of the reviewed related works covers all these characteristics to the best of the authors’ knowledge. In addition, they are incorporated while scoring error metrics similar to or better than the baselines. Specifically, normalized Root-Mean-Square Error improves 14.1%, 0.8%, and 6.8% compared to the persistence, three-dimensional convolutional, and Long Short-Term Memory models, respectively.

Different accurate predictive models have been developed to forecast the amount of solar energy produced in a given area. These models are usually run in a centralized manner, considering irradiance inputs taken from a set of sensors that are deployed in that area. CAIDE is a framework that supports the deployment and analysis of solar plants following Model Based System Engineering (MBSE) and Internet of Things (IoT) methodologies. However, the current solution performs the training and inference phases of the solar energy forecasting models in a central way, not taking advantage of the distributed environment modeled by means of CAIDE. This work presents an extension of CAIDE that allows us to distribute the training and inference phases, obtaining performance improvements, and achieving a greater adaptation to the inherently distributed topology of the deployment of the sensors.

Ontology-based data access focuses on enabling query evaluation over heterogeneous relational databases according to the model represented by an ontology. The relationships between the ontology and the data sources are commonly defined with declarative mappings, which are used by systems to perform SPARQL-to-SQL query translation or to generate RDF dumps from the relational databases. Besides the potential homogenization of data because of using an ontology, some additional advantages of this paradigm are that it may allow applying reasoning thanks to the ontology, as well as querying for meta knowledge, which describes statements with information such as provenance or certainty. In this paper, (i) we adapt a widely used RDF graph store benchmark, namely LUBM, for ontology-based data access, (ii) extend the benchmark for the evaluation of queries that exploit meta knowledge, and (iii) apply it for performance evaluation of state-of-the-art declarative mapping systems. Our proposal, the LUBM4OBDA Benchmark, considers inference capabilities that are not covered by previous ontology-based data access benchmarks, and it is the first one for the evaluation of meta knowledge and the RDF-star data model. The experimental evaluation shows that current virtualization systems cannot handle some advanced inference tasks, and that optimizations are needed to scale RDF-star materialization.

Small files are known to pose major performance challenges for file systems. Yet, such workloads are increasingly common in a number of Big Data Analytics workflows or large-scale HPC simulations. These challenges are mainly caused by the common architecture of most state-of-the-art file systems needing one or multiple metadata requests before being able to read from a file. Small input file size causes the overhead of this metadata management to gain relative importance as the size of each file decreases. In this paper we propose a set of techniques leveraging consistent hashing and dynamic metadata replication to significantly reduce this metadata overhead. We implement such techniques inside a new file system named TýrFS, built as a thin layer above the Týr object store. We prove that TýrFS increases small file access performance up to one order of magnitude compared to other state-of-the-art file systems, while only causing a minimal impact on file write throughput.

Big Data applications are increasingly moving from batch-oriented execution models to stream-based models that enable them to extract valuable insights close to real-time. To support this model, an essential part of the streaming processing pipeline is data ingestion, i.e., the collection of data from various sources (sensors, NoSQL stores, filesystems, etc.) and their delivery for processing. Data ingestion needs to support high throughput, low latency and must scale to a large number of both data producers and consumers. Since the overall performance of the whole stream processing pipeline is limited by that of the ingestion phase, it is critical to satisfy these performance goals. However, state-of-art data ingestion systems such as Apache Kafka build on static stream partitioning and offset-based record access, trading performance for design simplicity. In this paper we propose KerA, a data ingestion framework that alleviate the limitations of state-of-art thanks to a dynamic partitioning scheme and to lightweight indexing, thereby improving throughput, latency and scalability. Experimental evaluations show that KerA outperforms Kafka up to 4x for ingestion throughput and up to 5x for the overall stream processing throughput. Furthermore, they show that KerA is capable of delivering data fast enough to saturate the big data engine acting as the consumer.

Cloud developers traditionally rely on purpose-specific services to provide the storage model they need for an application. In contrast, HPC developers have a much more limited choice, typically restricted to a centralized parallel file system for persistent storage. Unfortunately, these systems often offer low performance when subject to highly concurrent, conflicting I/O patterns. This makes difficult the implementation of inherently concurrent data structures such as distributed shared logs. Yet, this data structure is key to applications such as computational steering, data collection from physical sensor grids, or discrete event generators. In this paper we tackle this issue. We present SLoG, shared log middleware providing a shared log abstraction over a parallel file system, designed to circumvent the aforementioned limitations. We evaluate SLoG's design on up to 100,000 cores of the Theta supercomputer: the results show high append velocity at scale while also providing substantial benefits for other persistent backend storage systems.

Linked Data repositories have become a popular source of publicly-available data. Users accessing this data through SPARQL endpoints usually launch several restrictive yet similar consecutive queries, either to find the information they need through trial-and-error or to query related resources. However, instead of executing each individual query separately, query augmentation aims at modifying the incoming queries to retrieve more data that is potentially relevant to subsequent requests. In this paper, we propose a novel approach to query augmentation for SPARQL endpoints based on machine learning. Our approach separates the structure of the query from its contents and measures two types of similarity, which are then used to predict the structure and contents of the augmented query. We test the approach on the real-world query logs of the Spanish and English DBpedia and show that our approach yields high-accuracy prediction. We also show that, by caching the results of the predicted (More)

Monitoring has always been a key element on ensuring the performance of complex distributed systems, being a first step to control quality of service, detect anomalies, or make decisions about resource allocation and job scheduling, to name a few. Edge computing is a new type of distributed computing, where data processing is performed by a large number of heterogeneous devices close to the place where the data is generated. Some of the differences between this approach and more traditional architectures, like cloud or high performance computing, are that these devices have low computing power, have unstable connectivity, and are geo‐distributed or even mobile. All of these aforementioned characteristics establish new requirements for monitoring tools, such as customized monitoring workflows or choosing different back‐ends for the metrics, depending on the device hosting them. In this paper, we present a study of the requirements that an edge monitoring tool should meet, based on motivating scenarios drawn from literature. Additionally, we implement these requirements in a monitoring tool named FMonE. This framework allows deploying monitoring workflows that conform to the specific demands of edge computing systems. We evaluate FMonE by simulating a fog environment in the Grid’5000 testbed and we demonstrate that it fulfills the requirements we previously enumerated.

Big Data applications are rapidly moving from a batch-oriented execution model to a streaming execution model in order to extract value from the data in real-time. However, processing live data alone is often not enough: in many cases, such applications need to combine the live data with previously archived data to increase the quality of the extracted insights. Current streaming-oriented runtimes and middlewares are not flexible enough to deal with this trend, as they address ingestion (collection and pre-processing of data streams) and persistent storage (archival of intermediate results) using separate services. This separation often leads to I/O redundancy (e.g., write data twice to disk or transfer data twice over the network) and interference (e.g., I/O bottlenecks when collecting data streams and writing archival data simultaneously). In this position paper, we argue for a unified ingestion and storage architecture for streaming data that addresses the aforementioned challenge. We identify a set of constraints and benefits for such a unified model, while highlighting the important architectural aspects required to implement it in real life. Based on these aspects, we briefly sketch our plan for future work that develops the position defended in this paper.

International audience

Big Data analytics has recently gained increasing popularity as a tool to process large amounts of data on-demand. Spark and Flink are two Apache-hosted data analytics frameworks that facilitate the development of multi-step data pipelines using directly acyclic graph patterns. Making the most out of these frameworks is challenging because efficient executions strongly rely on complex parameter configurations and on an in-depth understanding of the underlying architectural choices. Although extensive research has been devoted to improving and evaluating the performance of such analytics frameworks, most of them benchmark the platforms against Hadoop, as a baseline, a rather unfair comparison considering the fundamentally different design principles. This paper aims to bring some justice in this respect, by directly evaluating the performance of Spark and Flink. Our goal is to identify and explain the impact of the different architectural choices and the parameter configurations on the perceived end-to-end performance. To this end, we develop a methodology for correlating the parameter settings and the operators execution plan with the resource usage. We use this methodology to dissect the performance of Spark and Flink with several representative batch and iterative workloads on up to 100 nodes. Our key finding is that there none of the two framework outperforms the other for all data types, sizes and job patterns. This paper performs a fine characterization of the cases when each framework is superior, and we highlight how this performance correlates to operators, to resource usage and to the specifics of the internal framework design.

Concurrent Big Data applications often require high-performance storage, as well as ACID (Atomicity, Consistency, Isolation, Durability) transaction support. Although blobs (binary large objects) are an increasingly popular storage model for such applications, state-of-the-art blob storage systems offer no transaction semantics. This demands users to coordinate data access carefully in order to avoid race conditions, inconsistent writes, overwrites and other problems that cause erratic behavior. We argue there is a gap between existing storage solutions and application requirements, which limits the design of transaction-oriented applications. We introduce Týr, the first blob storage system to provide built-in, multiblob transactions, while retaining sequential consistency and high throughput under heavy access concurrency. Týr offers fine-grained random write access to data and in-place atomic operations. Large-scale experiments with a production application from CERN LHC show Týr throughput outperforming state-of-the-art solutions by more than 75%.

Apache Cassandra is an open-source cloud storage system that offers multiple types of operation-level consistency including eventual consistency with multiple levels of guarantees and strong consistency. It is being used by many data-center applications (e.g., Facebook and App Scale). Most existing research efforts have been dedicated to exploring trade-offs such as: consistency vs. Performance, consistency vs. Latency and consistency vs. Monetary cost. In contrast, a little work is focusing on the consistency vs. Energy trade-off. As power bills have become a substantial part of the monetary cost for operating a data-center, this paper aims to provide a clearer understanding of the interplay between consistency and energy consumption. Accordingly, a series of experiments have been conducted to explore the implication of different factors on the energy consumption in Cassandra. Our experiments have revealed a noticeable variation in the energy consumption depending on the consistency level. Furthermore, for a given consistency level, the energy consumption of Cassandra varies with the access pattern and the load exhibited by the application. This further analysis indicates that the uneven distribution of the load amongst different nodes also impacts the energy consumption in Cassandra. Finally, we experimentally compare the impact of four storage configuration and data partitioning policies on the energy consumption in Cassandra: interestingly, we achieve 23% energy saving when assigning 50% of the nodes to the hot pool for the applications with moderate ratio of reads and writes, while applying eventual (quorum) consistency. This study points to opportunities for future research on consistency-energy trade-offs and offers useful insight into designing energy-efficient techniques for cloud storage systems.

With the emergence of cloud computing, many organizations have moved their data to the cloud in order to provide scalable, reliable and highly available services. To meet the ever-growing user needs, these services mainly rely on geographically-distributed data replication to guarantee good performance and high availability. However, with replication, consistency comes into question. Service providers in the cloud have the freedom to select the level of consistency according to the access patterns exhibited by the applications. Most optimizations efforts then concentrate on how to provide adequate trade-offs between consistency guarantees and performance. However, as the monetary cost completely relies on the service providers, in this paper we argue that monetary cost should be taken into consideration when evaluating or selecting a consistency level in the cloud. Accordingly, we define a new metric called consistency-cost efficiency. Based on this metric, we present a simple, yet efficient economical consistency model, called Bismar, that adaptively tunes the consistency level at runtime in order to reduce the monetary cost while simultaneously maintaining a low fraction of stale reads. Experimental evaluations with the Cassandra cloud storage on the Grid'5000 test bed show the validity of the metric and demonstrate the effectiveness of the proposed consistency model.

Cloud computing is one the most relevant computing paradigms available nowadays. Its adoption has increased during last years due to the large investment and research from business enterprises and academia institutions. Among all the services cloud providers usually offer, Infrastructure as a Service has reached its momentum for solving HPC problems in a more dynamic way without the need of expensive investments. The integration of a large number of providers is a major goal as it enables the improvement of the quality of the selected resources in terms of pricing, speed, redundancy, etc. In this paper, we propose a system architecture, based on semantic solutions, to build an interoperable scheduler for federated clouds that works with several IaaS (Infrastructure as a Service) providers in a uniform way. Based on this architecture we implement a proof-of-concept prototype and test it with two different cloud solutions to provide some experimental results about the viability of our approach.

In just a few years cloud computing has become a very popular paradigm and a business success story, with storage being one of the key features. To achieve high data availability, cloud storage services rely on replication. In this context, one major challenge is data consistency. In contrast to traditional approaches that are mostly based on strong consistency, many cloud storage services opt for weaker consistency models in order to achieve better availability and performance. This comes at the cost of a high probability of stale data being read, as the replicas involved in the reads may not always have the most recent write. In this paper, we propose a novel approach, named Harmony, which adaptively tunes the consistency level at run-time according to the application requirements. The key idea behind Harmony is an intelligent estimation model of stale reads, allowing to elastically scale up or down the number of replicas involved in read operations to maintain a low (possibly zero) tolerable fraction of stale reads. As a result, Harmony can meet the desired consistency of the applications while achieving good performance. We have implemented Harmony and performed extensive evaluations with the Cassandra cloud storage on Grid'5000 test bed and on Amazon EC2. The results show that Harmony can achieve good performance without exceeding the tolerated number of stale reads. For instance, in contrast to the static eventual consistency used in Cassandra, Harmony reduces the stale data being read by almost 80% while adding only minimal latency. Meanwhile, it improves the throughput of the system by 45% while maintaining the desired consistency requirements of the applications when compared to the strong consistency model in Cassandra.

Complexity has always been one of the most important issues in distributed computing. From the first clusters to grid and now cloud computing, dealing correctly and efficiently with system complexity is the key to taking technology a step further. In this sense, global behavior modeling is an innovative methodology aimed at understanding the grid behavior. The main objective of this methodology is to synthesize the grid's vast, heterogeneous nature into a simple but powerful behavior model, represented in the form of a single, abstract entity, with a global state. Global behavior modeling has proved to be very useful in effectively managing grid complexity but, in many cases, deeper knowledge is needed. It generates a descriptive model that could be greatly improved if extended not only to explain behavior, but also to predict it. In this paper we present a prediction methodology whose objective is to define the techniques needed to create global behavior prediction models for grid systems. This global behavior prediction can benefit grid management, specially in areas such as fault tolerance or job scheduling. The paper presents experimental results obtained in real scenarios in order to validate this approach.

The cloud computing model aims to make large-scale data-intensive computing affordable even for users with limited financial resources, that cannot invest into expensive infrastructures necesssary to run them. In this context, MapReduce is emerging as a highly scalable programming paradigm that enables high-throughput data-intensive processing as a cloud service. Its performance is highly dependent on the underlying storage service, responsible to efficiently support massively parallel data accesses by guaranteeing a high throughput under heavy access concurrency. In this context, quality of service plays a crucial role: the storage service needs to sustain a stable throughput for each individual accesss, in addition to achieving a high aggregated throughput under concurrency. In this paper we propose a technique to address this problem using component monitoring, application-side feedback and behavior pattern analysis to automatically infer useful knowledge about the causes of poor quality of service and provide an easy way to reason in about potential improvements. We apply our proposal to Blob Seer, a representative data storage service specifically designed to achieve high aggregated throughputs and show through extensive experimentation substantial improvements in the stability of individual data read accesses under MapReduce workloads.

Grid systems have proved to be one of the most important new alternatives to face challenging problems but, to exploit its benefits, dependability and fault tolerance are key aspects. However, the vast complexity of these systems limits the efficiency of traditional fault tolerance techniques. It seems necessary to distinguish between resource-level fault tolerance (focused on every machine) and service-level fault tolerance (focused on global behavior). Techniques based on these concepts can handle system complexity and increase dependability. We present an autonomous, self-adaptive fault tolerance framework for grid systems, based on a new approach to model distributed environments. The grid is considered as a single entity, instead of a set of independent resources. This point of view focuses on service-level fault tolerance, allowing us to see the big picture and understand the system's global behavior. The resulting model's simplicity is the key to provide system-wide fault tolerance.

Abstract Over the last decade, grid computing has paved the way for a new level of large‐scale‐distributed systems. However, this new step in distributed computing comes along with a completely new level of complexity. Grid management mechanisms play a key role, and a correct analysis and understanding of the grid behavior is needed. Traditional‐distributed computing management mechanisms analyze each resource separately and adjust specific parameters of each one of them. When trying to adapt the same procedures to grid computing, the vast complexity of the system can complicate this task. But grid complexity could only be a matter of perspective. It is possible to understand the grid behavior as a single system, instead of a set of resources. This abstraction could provide a deeper understanding of the system, describing large‐scale behavior and global events that probably would not be detected while analyzing each resource separately. In this paper a specific methodology is presented and described in order to create a global behavior model of the grid, analyzing it as a single entity. Both real and simulated case studies are also presented, in order to provide a proper validation and illustrate the benefits of this approach. Copyright © 2009 Crown in the right of Canada. Published by John Wiley & Sons, Ltd.

In the recent decades we have witnessed a major revolution in the computer field. The major challenges posed by applications in fields of bioinformatics, earth sciences or weather forecasting, among others, have caused the proliferation of complex solutions, such as grid, cloud and highperformance computing. The common objective of all these disciplines is the sharing of hardware and software resources to provide an infrastructure in which to run efficiently these applications. Particularly, grid computing has been one of the most important computing topics in the last years. Within this context, the GADA workshop arose in 2004 as a forum for researchers in grid computing and its application to data analysis. From then until 2008, GADA became a reference conference for researchers in grid, covering also a broader set of disciplines, although grid computing continued to play a key role in the set of main topics of the conference. This special issue includes the eight best papers presented at the International Conference on Grid Computing, High Performance and Distributed Application (GADA 2008) from a total of 31 submissions with an acceptance rate of 26%. GADA 2008 was held in conjunction with the On The Move Conferences during November 2008 in Monterrey, Mexico. Each submitted paper was reviewed by three reviewers and one of the program chairs, and a total of 70 reviewers were involved in the review process of GADA 2008. A further review stage was performed to select the papers for this special issue. Topics covered by these papers include grid modelling, performance and scalability. Three different papers address the important topic of grid modelling. Branco et al. [1] describe the experience in the development of the data management middleware DQ2, used in the ATLAS experiment for the Large Hadron Collider (HLC). From this experience, they have identified an important degree of uncertainty over the behaviour of large grid infrastructures. From the analysis of this uncertainty, they propose novel modelling and simulation techniques for Data Grids. van der Aalst et al. [2] show a formal description of the grid in terms of a Colored Petri Net (CPN). This formalism can be used as a conceptual model of a grid environment, and also allows you to perform various types of analysis, including performance analysis. The model has been validated by means of experiments in a testbed grid architecture. Montes et al. [3] present a methodology based on data mining techniques for the building of a global behaviour model of the grid. This methodology deals with the complexity of grid environments, providing a simple model that can be used for administrative purposes. The validation of the model has been performed by means of real and simulated case studies.

The use of a Semantic Grid architecture can ease the deployment of complex applications, in which several organizations are involved and where resources of diverse nature (data and computing elements) are shared. This is the situation in the Space domain, with a strong demand of computational resources inscribed in an extensive and heterogeneous network of facilities and institutions. This paper presents the S-OGSA architecture, defined in the Ontogrid project, as applied into a scenario for the overall monitoring and data analysis in a Satellite Mission currently in nominal operations. Flexibility, scalability, interoperability and use of a common framework for data sharing are the main advantages of a Semantic Grid implementation in Complex and dataintensive systems.

The TV environment is not so different from any other grid computing scenario, as different TV channels can need to share multimedia information and resources to achieve some specific purposes on time. In this paper, we present a web service specification to manage awareness in collaborative grid environments, WS-AMBLE. This specification has been designed by merging web services with multi-agents theories and principles to provide an autonomous, efficient and independent management of the amount of resources available in TV environment. WSAMBLE implementation makes easier the collaboration and cooperation among different TV channels. In this paper, we also present some experimental results carried out over a real and heterogeneous grid environment with the end of emphasizing the performance improvements that WS-AMBLE has in a Grid-TV environment.

The convergence of the Semantic Web and Grid technologies has resulted in the Semantic Grid. The Semantic Grid should be service-oriented, as the Grid is, so the formal description of Grid Services (GS) turns to be a crucial issue. In this paper we present our approach for this issue. ODESGS Framework will enable the annotation of all the aspects of a GS and the design, discovery and composition Semantic Grid Services (SGS).

MAPFS is a parallel file system integrated with a multiagent system responsible for the information retrieval. One of the fields where the agents can be very useful is precisely in the development of information recovery systems. The usage of a multiagent system implies coordination among the agents that belong to such system. The main goal of the agent cooperation is the interaction among them for achieving a common objective in a distributed system. Thus, a communication framework must be provided. This paper shows the MAPFS cooperation model and its communication framework, emphasizing its relation with the whole system.

The WIC(Web Intelligence Consortium)-Spain Research Centre is an interdisciplinary research initiative which brings together people with the aim of covering all the aspects related to the discovery of hidden knowledge in big volumes of data as long as exploring the practical impacts of advanced Information Technology (IT) and Artificial Intelligence (AI) on this volume of information The group consist of seven researchers and five Ph.D. students led by Dr. Javier Segovia and coordinated by Dr. Ernestina Menasalvas. We are funded from a variety of sources including the Spanish Ministry of Science and Technology as well as national and international industry.

Recommender systems emerged to help users choose among the large amount of options that ecommerce sites offer. Collaborative filtering is one of the most successful recommender techniques. Here we propose an approach to collaborative filtering based on the simple Bayesian classifier. We propose a method of increasing the efficiency of naive Bayes by applying a new semi naive Bayes approach based on interval estimation. To evaluate our algorithm we use a database of Microsoft anonymous Web data from the UCl repository. Our empirical results show that our proposed Interval based naive Bayes approach outperforms typical naive Bayes.

MAPFS is a parallel file system integrated with a multiagent system responsible for the information retrieval [2]. The use of a multiagent system implies coordination among the agents such system consists of. The principal