Summary This article introduces a new library that leverages and extends the MPI (message passing interface) standard, capable of aggregating servers and clusters located in multiple domains, as well as cloud resources. The execution platform is created and presented to the MPI application transparently, without the need of recompiling any code. The solution also provides functions for provisioning, adding, and removing nodes at runtime, bringing elasticity to the application. Through benchmarks, its performance was compared with the native execution of the application using the native MPI library. A prototype of an elastic application was also developed, with positive results within the expected range.

With the exponential volume growth of the data available for scientific and commercial use, more and more Big Data technologies are gaining focus and importance. Directly related to the efficiency of these techniques is the distributed file system used for data persistence, generally based on low-cost computer clusters. However, the environments used today for Big Data are based on file systems restricted to the WORM pattern (write once, read many) lacking POSIX compatibility. This work uses distributed lock management techniques to create a file system that allows random writing for both HPC and Big Data tools. A local write protocol is implemented to leverage the use of local copies of the data during the write process. Experiments were carried out to evaluate the performance of the proposed write protocol and the scalability of the developed file system. From the experimental results, it is possible to conclude that the achieved performance and scalability improvements were obtained by eliminating limitations imposed by HDFS and leveraging local writes.

Summary Data abundance poses the need for powerful and easy‐to‐use tools that support processing large amounts of data. MapReduce has been increasingly adopted for over a decade by many companies, and more recently, it has attracted the attention of an increasing number of researchers in several areas. One main advantage is that the complex details of parallel processing, such as complex network programming, task scheduling, data placement, and fault tolerance, are hidden in a conceptually simple framework. MapReduce is supported by mature software technologies for deployment in data centers such as Hadoop. As MapReduce becomes popular for high‐performance applications, many questions arise concerning its performance and efficiency. In this paper, we demonstrated formally lower bounds on the isoefficiency function for MapReduce applications, when these applications can be modeled as BSP jobs. We also demonstrate how communication and synchronization costs can be dominant for MapReduce computations and discuss the conditions under which such scalability limits are valid. To our knowledge, this is the first study that demonstrates scalability bounds for MapReduce applications. We also discuss how some MapReduce implementations such as Hadoop can mitigate such costs to approach linear, or near‐to‐linear speedups. Copyright © 2015 John Wiley & Sons, Ltd.

The volume of data stored in corporate data centers has been growing at a rate of 35% to 50% per year [1]. The exponential growth in data volume leads to some challenges from the technical, operational and financial perspectives. Along with this increase in the data volume the demand for preservation (or retention) of such data has also increased due to government regulations. The convergence of these two trends (growth of data volume and increased demand for preservation) implies that storage systems must support the preservation of data for very long periods of time. Several studies address the archiving, preservation and retrieval of structured data. To the best of our knowledge, there couldn't be found reference architectures specifically focused on the archiving, preservation and retrieval of both unstructured and structured data. Our research goal is to propose a reference architecture for the long term archiving, preservation and retrieval of Big Data.

Databases are normally designed to support a specific application domain, and the globalization process brings new issues to integrate data from more than one database. This paper proposes an architecture to integrating databases, by demonstrating a method for integrating semantically heterogeneous databases with a single tool and grid computing technology, which enables access to different databases in different formats through a single interface, as if they were a single database. Thus, this architecture can carry out queries simultaneously on shared integrated databases in a parallel way.

The low cost and wide availability of PC-based clusters have made them an excellent alternative to access supercomputing. However, while network of workstations may be readily available, there is an increasing need for performance tools that support these platforms, in order to achieve even higher performance. One of possible ways to increase performance is parallel program restructuring. It is introduced in this paper a toolkit to generate graphical charts for visualization of MPI parallel programs, reflecting to its execution over time, with the use of DP*Graph representation, parallel version of timing graph. In other words, parallel programs are shown through charts its sequential codes, dependencies and communication structures in a particular cluster system platform. Still in this paper, it is discussed the implementation of this toolkit and present some experimental results obtained.

This paper presents the HMPI, a runtime system to integrate several MPI implementations, used to develop high performance applications that must run both in nodes with several operating systems and clusters of clusters infrastructures. HMPI has two different approaches to achieve this integration, using connection pools and HMPI daemons.

The advances in hardware technology enable the inclusion of an SMP node into a cluster of PC or even clusters of SMP. These are becoming viable alternatives for high performance computing. The challenge is the exploration of the computational resources that these hardware platforms provide. A hybrid programming paradigm which uses a shared memory architecture through multi threading and through a message passing model for inter node communication is an alternative. However, programming in such a paradigm is very hard. This work presents CPAR-Cluster, a runtime system that provides shared memory abstraction on top of a cluster composed by mono and multiprocessor nodes. Its implementation is at the library level and does not require special resources such as specific hardware or operating system modifications. Models, strategies, implementation aspects and some results are presented.

This paper discusses a load distribution strategy for distributed object computing systems. Our strategy relies on dynamic monitoring and application profiling to manage workloads and improve the system performance of non-dedicated clusters of heterogeneous computers. Application profiling and the combined use of load balancing mechanisms have proven useful to deal with services with a differentiated demand for resources. The use of this strategy allows for load balancing and performance improvement with low overhead.

This software version is a distributed implementation of the public domain software of groundwater dispersion simulation of Anderson's book. This simulator proposed in Anderson's book is a sequential program that calculates the concentrations during the dispersion of an aquifer. Basically, the numerical method used in this simulator is finite differences. Depending on the number of NNODEs (number of nodes in the problem domain) used in the grid for the simulation, the simulation time can be prohibitive. The main subject of this paper is to evaluate the performance of a new version of DGWSDS (Distributed Groundwater Simulation Dispersion System). Running the distributed simulator with three different input parameters size NN-ODE=2000, 3000, 4000 and 5000, for a cluster of 8 nodes, the DGWSDS achieves speedups up to 10.73, which is an excellent value if the interconnection and the overhead of the TCP/IP are considered.