Stream computing is becoming a more and more popular paradigm as it enables the real-time promise of data analytics. Apache Kafka is currently the most popular framework used to ingest the data streams into the processing platforms. However, how to tune Kafka and how much resources to allocate for it remains a challenge for most users, who now rely mainly on empirical approaches to determine the best parameter settings for their deployments. In this poster, we make a through evaluation of several configurations and performance metrics of Kafka in order to allow users avoid bottlenecks, reach its full potential and avoid bottlenecks and eventually leverage some good practice for efficient stream processing.

This special issue of Concurrency and Computation: Practice and Experience contains revised and extended versions of selected papers presented at the conference Euro-Par 2016.Euro-Par-the European Conference on Parallel Computing-is an annual series of international conferences dedicated to the promotion and advancement of all aspects of parallel and distributed computing.Euro-Par covers a wide spectrum of topics from algorithms and theory to software technology and hardware-related issues, with application areas ranging from scientific to mobile and cloud computing.The major part of the Euro-Par audience consists of researchers in academic institutions, government laboratories, and industrial organisations.Euro-Par 2016, the 22nd conference in the Euro-Par series, was held in Grenoble, France.It was organised by Inria, Université Grenoble-Alpes, and IUT 2 Grenoble.Twelve broad topics were defined and advertised, covering a large variety of aspects of parallel and distributed computing.The call for papers attracted a total of 176 submissions.The submitted papers were reviewed at least 3 and, in most cases, 4 or even more times (4 reviews on average).A total of 47 papers were finally accepted for publication.This makes a global acceptance rate of 26.7 %.The authors of accepted papers came from 20 countries, with the 4 main contributing countries-France, the United States, Germany, and Spain-accounting for a bit more than half of them.Compared to the conference version, this framework is enhanced further with the availability of customized CUDA kernels and a multiple-GPU implementation with almost linear scalability.The reviewers appreciated the high quality and scientific soundness of the treatise.Concluding this preface, we would like to thank Prof Geoffrey Fox, editor-in-chief of Concurrency and Computation: Practice and Experience, for his support of this special issue.We

While many parallel visualization tools now provide in situ visualization capabilities, the trend has been to feed such tools with large amounts of unprocessed output data and let them render everything at the highest possible resolution. This leads to an increased run time of simulations that still have to complete within a fixed-length job allocation. In this paper, we tackle the challenge of enabling in situ visualization under performance constraints. Our approach shuffles data across processes according to its content and filters out part of it in order to feed a visualization pipeline with only a reorganized subset of the data produced by the simulation. Our framework leverages fast, generic evaluation procedures to score blocks of data, using information theory, statistics, and linear algebra. It monitors its own performance and adapts dynamically to achieve appropriate visual fidelity within predefined performance constraints. Experiments on the Blue Waters supercomputer with the CM1 simulation show that our approach enables a 5x speedup with respect to the initial visualization pipeline and is able to meet performance constraints.

This special issue of Concurrency and Computation: Practice and Experience contains revised and extended versions of selected papers presented at the conference Euro-Par 2015.Euro-Par-the European Conference on Parallel Computing-is an annual series of international conferences dedicated to the promotion and advancement of all aspects of parallel and distributed computing.Euro-Par covers a wide spectrum of topics from algorithms and theory to software technology and hardware-related issues, with application areas ranging from scientific to mobile and cloud computing.The major part of the Euro-Par audience consists of researchers in academic institutions, government laboratories and industrial organisations.Euro-Par 2015, the 21st conference in the Euro-Par series, was held in Vienna, Austria.It was organised by the Research Group for Parallel Computing of the Vienna University of Technology (TU Wien).Thirteen broad topics were defined and advertised, covering a large variety of aspects of parallel and distributed computing.The call for papers attracted a total of 190 submissions.The submitted papers were reviewed at least three and, in most cases, four or even more times (four reviews on average).A total of 51 papers were finally accepted for publication.This makes a global acceptance rate of 27 %.The authors of accepted papers came from 21 countries, with the four main contributing countries-the United States, France, Spain and Germany-accounting for a bit more than half of them.Based on the results of the reviews and a majority opinion of the respective topic programme committees, a number of papers were recommended for this special issue.The authors were contacted at the conference and invited to submit revised and extended versions of their papers.These new versions were reviewed independently by three reviewers; two had previously reviewed the conference version, the third had not.Eventually, four papers were accepted for publication.This year, two Euro-Par topics are represented-both covering methods of programming modern computer architectures.Topic 13 on Accelerator Computing is represented with three papers.The paper Performance optimization of sparse matrix-vector multiplication for multi-component PDE-based applications using GPUs, authored by Ahmad Abdelfattah, Hatem Ltaief, David Keyes and Jack Dongarra [1], describes the implementation of a single-GPU and multi-GPU kernel for block-sparse matrix-vector multiplication, a problem that appears in the discretisation of partial differential equations with many dependent variables.The performance of the kernel is measured on a subset of the Florida Sparse Matrix Collection.Especially noted by the reviewers was the uniform interface that applies to a wide range of problem sizes via tunable parameters.This makes it perform efficiently on a wide range of GPU architectures running CUDA.The paper Fast parallel skew and prefix-doubling suffix array construction on the GPU, authored by Leyuan Wang, Sean Baxter and John D. Owens [2], proposes a hybrid GPU implementation of known algorithms for constructing suffix arrays of a string that fits the given GPU architecture best.One highlight pointed out in the reviews is a highly efficient segmented sorting primitive, which is also valuable as independent result.The paper Performance and portability of accelerated lattice Boltzmann applications with OpenACC, authored by Enrico Calore, Jiri Kraus, Sebastiano Fabio Schifano and Raffaele Tripiccione [3], reports on a performance study based on a simple performance model of an OpenACCbased lattice Boltzmann implementation on three different architectures: an NVIDIA GPU, an AMD GPU and a multi-core CPU.The practical relevance of this work was particularly appreciated.

Apache Cassandra is an open-source cloud storage system that offers multiple types of operation-level consistency including eventual consistency with multiple levels of guarantees and strong consistency. It is being used by many data-center applications (e.g., Facebook and App Scale). Most existing research efforts have been dedicated to exploring trade-offs such as: consistency vs. Performance, consistency vs. Latency and consistency vs. Monetary cost. In contrast, a little work is focusing on the consistency vs. Energy trade-off. As power bills have become a substantial part of the monetary cost for operating a data-center, this paper aims to provide a clearer understanding of the interplay between consistency and energy consumption. Accordingly, a series of experiments have been conducted to explore the implication of different factors on the energy consumption in Cassandra. Our experiments have revealed a noticeable variation in the energy consumption depending on the consistency level. Furthermore, for a given consistency level, the energy consumption of Cassandra varies with the access pattern and the load exhibited by the application. This further analysis indicates that the uneven distribution of the load amongst different nodes also impacts the energy consumption in Cassandra. Finally, we experimentally compare the impact of four storage configuration and data partitioning policies on the energy consumption in Cassandra: interestingly, we achieve 23% energy saving when assigning 50% of the nodes to the hot pool for the applications with moderate ratio of reads and writes, while applying eventual (quorum) consistency. This study points to opportunities for future research on consistency-energy trade-offs and offers useful insight into designing energy-efficient techniques for cloud storage systems.

This special issue of Concurrency and Computation: Practice and Experience contains revised and extended versions of selected papers presented at the conference Euro-Par 2014.Euro-Par-the European Conference on Parallel Computing-is an annual series of international conferences dedicated to the promotion and advancement of all aspects of parallel and distributed computing.Euro-Par covers a wide spectrum of topics from algorithms and theory to software technology and hardware-related issues, with application areas ranging from scientific to mobile and cloud computing.The major part of the Euro-Par audience consists of researchers in academic institutions, government laboratories and industrial organisations.Euro-Par 2014, the 20th conference in the Euro-Par series, was held in Porto, Portugal.It was organised by the Computer Science Department of the Faculty of Sciences of the University of Porto.Fifteen broad topics were defined and advertised, covering a large variety of aspects of parallel and distributed computing.The call for papers attracted a total of 267 submissions.The submitted papers were reviewed at least three, and in most cases, four, times (4.02 on average).A total of 68 papers were finally accepted for publication.This makes a global acceptance rate of 25.5%.The authors of accepted papers came from 29 countries, with the four main contributing countries-France, the United States, Spain and Germany-accounting for about 55% of them.Based on the results of the reviews and a majority opinion of the respective topic programme committees, several papers were recommended for a special journal issue.The authors were contacted at the conference and invited to submit revised and extended versions of their papers.These new versions were reviewed independently by three reviewers; two had previously reviewed the conference version, the third had not.Eventually, four papers were accepted for publication.They cover the following four Euro-Par topics: Performance Prediction and Evaluation, Distributed Systems and Algorithms, Theory and Algorithms for Parallel Computation and High-Performance and Scientific Applications.Topic 2 on Performance Prediction and Evaluation contributes the paper Performance prediction of dynamic task-based runtime system for heterogeneous multi-core architectures authored by Luka Stanisic, Samuel Thibault, Arnaud Legrand, Brice Videau and Jean-François Méhaut [1].They address the challenge of deciding, with high accuracy and low effort, what computations to offload onto accelerators on a heterogeneous execution platform.Their coarse-grain hybrid simulation/emulation of StarPU ‡ on SimGrid, a versatile simulator for distributed systems, yields performance predictions of dense linear algebra applications in a matter of seconds and with an accuracy of within a few percent.The reviewers appreciated particularly the volume of experimental results and analysis on a wide range of heterogeneous platforms.Topic 8 on Distributed Systems and Algorithms is represented by the paper A comparative study of spanning tree and gossip protocols for aggregation authored by Lehel Nyers and Márk Jelasity [2].The authors provide a carefully designed comparative study of two paradigms for distributed aggregation queries, like average and sum: gossiping and tree algorithms.With a demonstration suite covering different practical topologies and scenarios, they identify the dominating distinguishing influences and do away with stereotypes such as 'gossip is slow and expensive' and 'trees are fragile and complicated'.The reviewers valued the experimental framework as an effective decision tool for choosing between the two paradigms.Topic 12 on Theory and Algorithms for Parallel Computation contributes the paper On constructing DAG-schedules with large AREAs authored by Scott T.

International audience

International audience

International audience

International audience

We consider the challenge of building data management systems that meet an important requirement of today's data-intensive HPC applications: to provide a high I/O throughput while supporting highly concurrent data accesses. In this context, many applications rely on MPI-I/O and require atomic, non-contiguous I/O operations that concurrently access shared data. In most existing implementations, the atomicity requirement is implemented through locking-based schemes, which have proven inefficient, especially for non-contiguous I/O. We claim that using a versioning-enabled storage back-end has the potential to avoid the expensive synchronization induced by locking-based schemes. We describe a prototype implementation on top of ROMIO, and report on promising experimental results with standard MPI-I/O benchmarks specifically designed to evaluate the performance of non-contiguous, overlapped I/O accesses under MPI atomicity guarantees.

Abstract Preface for the special issue on Euro‐Par 2009. Copyright © 2010 John Wiley & Sons, Ltd.

This special issue of Concurrency and Computation: Practice and Experience contains revised and extended versions of selected papers presented at the Euro-Par 2010 conference.Euro-Par-the European Conference on Parallel Computing-is an annual series of international conferences dedicated to the promotion and the advancement of all aspects of parallel and distributed computing.Euro-Par covers a wide spectrum of topics from algorithms and theory to software technology and hardware-related issues, with application areas ranging from scientific to mobile and cloud computing.The main audience of Euro-Par are the researchers in academic institutions, government laboratories, and industrial organizations.Euro-Par 2010, the 16th conference in the Euro-Par series, was organized by the Institute for High Performance Computing and Networking of the Italian National Research Council and was held at the Hotel Continental Terme on the Italian island of Ischia, in the Naples Bay.Fourteen broad topics were defined and advertised, covering a large variety of aspects of parallel and distributed computing.The call for papers attracted a total of 256 submissions.The submitted papers were reviewed at least three and, in many cases, four times.A total of 90 papers were finally accepted for publication.This makes a global acceptance rate of 35%.The authors of the accepted papers come from 24 countries, with the four main contributing countries-USA, France, Spain, and Germany-accounting for about 55% of the authors.The distribution of authors followed the pattern typical for a Euro-Par conference: of the 10 authors, this year, six have been academic researchers, three PhD students, and one from industry.The topical distribution of the papers in the proceedings of Euro-Par 2010 reflects the current trends in the field of parallel and distributed computing: 18 of 90 papers are devoted to the relatively new topic Multicore and Manycore Programming.This figure was 11 in 2009 and was even lower in 2008 when the subject was part of the topic High-Performance Architectures and Compilers.On the other hand, the well-established Euro-Par topics such as Support Tools and Environments or Scheduling and Load-Balancing, for instance, have remained quite stable throughout the years.These figures demonstrate that the topic structure of Euro-Par has the flexibility to adapt to current trends in a wide spectrum of research areas.With its topic structure, Euro-Par has been filling two complementary rôles: as a regular meeting place for established communities and as a place to develop new communities.In recent years, the latter rôle has been strengthened by the introduction of satellite workshops, at which researchers can meet initially and form a new community that later settles and flourishes in other fora.In 2010, the number of satellite workshops was higher than ever before, 11, and we strive to have it grow further, maintaining the Euro-Par claim of being the wide-spectrum conference on parallelism in Europe.Returning to the spread of papers in 2010, based on the results of the reviews and a majority opinion of the topic program committees, several papers were recommended for a special journal issue.The authors were contacted at the conference and invited to submit revised and extended versions of their papers.These new versions were reviewed independently by three reviewers; two had also reviewed the conference version, the third had not.Eventually, eight papers were accepted for publication.

Introspection is the prerequisite of an autonomic behavior, the first step towards a performance improvement and a resource-usage optimization for large-scale distributed systems. In grid environments, the task of observing the application behavior is assigned to monitoring systems. However, most of them are designed to provide general resource information and do not consider specific information for higher-level services. More specifically, in the context of data-intensive applications, a specific introspection layer is required in order to collect data about the usage of storage resources, about data access patterns, etc. This paper discusses the requirements for an introspection layer in a data-management system for large-scale distributed infrastructures. We focus on the case of BlobSeer, a large-scale distributed system for storing massive data. The paper explains why and how to enhance BlobSeer with introspective capabilities and proposes a three-layered architecture relying on the MonALISA monitoring framework. This approach has been evaluated on the Grid'5000 testbed, with experiments that prove the feasibility of generating relevant information related to the state and the behavior of the system.

Hadoop is a reference software framework supporting the Map/Reduce programming model. It relies on the Hadoop Distributed File System (HDFS) as its primary storage system. Although HDFS does not offer support for concurrently appending data to existing files, we argue that Map/Reduce applications as well as other classes of applications can benefit from such a functionality. We provide support for concurrent appends by building a concurrency-optimized data storage layer based on the BlobSeer data management service. Moreover, we modify the Hadoop Map/Reduce framework to use the append operation in the "reduce" phase of the application. To validate this work, we perform experiments on a large number of nodes of the Grid'5000 testbed. We demonstrate that massively concurrent append and read operations have a low impact on each other. Besides, measurements with an application available with Hadoop show that the support for concurrent appends to shared file is introduced with no extra cost, whereas the number of files managed by the Map/Reduced framework is substantially reduced.

Hadoop is a software framework supporting the Map-Reduce programming model. It relies on the Hadoop Distributed File System (HDFS) as its primary storage system. The efficiency of HDFS is crucial for the performance of Map-Reduce applications. We substitute the original HDFS layer of Hadoop with a new, concurrency-optimized data storage layer based on the BlobSeer data management service. Thereby, the efficiency of Hadoop is significantly improved for data-intensive Map-Reduce applications, which naturally exhibit a high degree of data access concurrency. Moreover, BlobSeer's features (built-in versioning, its support for concurrent append operations) open the possibility for Hadoop to further extend its functionalities. We report on extensive experiments conducted on the Grid'5000 testbed. The results illustrate the benefits of our approach over the original HDFS-based implementation of Hadoop.

Abstract Preface for the special issue on Euro‐Par 2007. Copyright © 2009 John Wiley & Sons, Ltd.

We consider the problem of efficiently managing massive data in a large-scale distributed environment. We consider data strings of size in the order of Terabytes, shared and accessed by concurrent clients. On each individual access, a segment of a string, of the order of Megabytes, is read or modified. Our goal is to provide the clients with efficient fine-grain access the data string as concurrently as possible, without locking the string itself. This issue is crucial in the context of applications in the field of astronomy, databases, data mining and multimedia. We illustrate these requirements with the case of an application for searching supernovae. Our solution relies on distributed, RAM-based data storage, while leveraging a DHT-based, parallel metadata management scheme. The proposed architecture and algorithms have been validated through a software prototype and evaluated in a cluster environment.

We consider the design of DSM consistency protocols for hierarchical architectures. Such architectures typically consist of a constellation of loosely-interconnected clusters, each cluster consisting of a set of tightly-interconnected nodes running multithreaded programs. We claim that high performance can only be reached by taking into account this interconnection hierarchy at the very core of the protocol design. Previous work has focused on improving locality in data management by caching remote data within clusters. In contrast, our idea is to improve locality in the synchronization management. We demonstrate the feasibility through an experimental implementation of this idea in a home-based protocol for Release Consistency, and we provide a preliminary evaluation of the expectable performance gain.

This paper introduces Madeleine II, a new adaptive and portable multi-protocol implementation of the Madeleine communication library. Madeleine II has the ability to control multiple network interfaces (BIP, SISCI, VIA) and multiple network adapters (Ethernet, Myrinet, SCI) within the same application session. Moreover it includes advanced mechanisms to dynamically select the most appropriate transfer method for a given network protocol according to various parameters such as data size or responsiveness user requirements. We report on performance measurements obtained using BIP/Myrinet and SISCI/SCI and we present preliminary results about our Nexus/Madeleine II and MPICH/Madeleine II ports.

We present an assertional approach to prove properties of ALPHA programs. ALPHA is a functional language based on affine recurrence equations. We first present two kinds of operational semantics for ALPHA together with some equivalence and confluence properties of these semantics. We then present an attempt to provide ALPHA with an external logical framework. We therefore define a proof method based on invariants. We focus on a particular class of invariants, namely canonical invariants, that are a logical expression of the program's semantics. We finally show that this framework is well-suited to prove partial properties, equivalence properties between ALPHA programs and properties that we cannot express within the ALPHA language.

A general definition of the notion of superimposition is presented. We show that previous constructions under the same name can be seen as special cases of our definition. We consider several properties of superimposition definable in our terms, notably the nonfreezing property. We also consider a syntactic representation of our construct in CSP