Hadoop is a popular open-source implementation of the MapReduce programming model for cloud computing. However, it faces a number of issues to achieve the best performance from the underlying system. These include a serialization barrier that delays the reduce phase, repetitive merges and disk access, and lack of capability to leverage latest high speed interconnects. We describe Hadoop-A, an acceleration framework that optimizes Hadoop with plugin components implemented in C++ for fast data movement, overcoming its existing limitations. A novel network-levitated merge algorithm is introduced to merge data without repetition and disk access. In addition, a full pipeline is designed to overlap the shuffle, merge and reduce phases. Our experimental results show that Hadoop-A doubles the data processing throughput of Hadoop, and reduces CPU utilization by more than 36%.

Sockets direct protocol (SDP) is a byte stream protocol that utilizes the capabilities of the InfiniBand fabric to transparently achieve performance gains for existing socket-based networked applications. We implemented SDP stack for the Windows operating system that is fully interoperable with Linux. The paper describes the early experience with the implementation of the protocol stack. We go through the motivation, the main implementation architectural aspects and challenges. We present preliminary performance results. Running over 20Gb/s InfiniBand double data rate (DDR) links we observed bandwidth record of 1316MB/S

Sockets Direct Protocol (SDP) is a byte stream protocol that utilizes the capabilities of the InfiniBand fabric to transparently achieve performance gains for existing socket-based networked applications. In this paper we discuss an implementation of Zero Copy support for synchronous send()/recv() socket calls, that uses the remote DMA capability of InfiniBand for SDP data transfers. We added this support to the open-source implementation of SDP over InfiniBand. We evaluate this implementation over a 20 Gb/s InfiniBand link. We demonstrate scalability of Zero Copy and show its benefits for systems that utilize multiple socket connections in parallel. For example, enabling Zero Copy with 8 active connections yields a bandwidth growth from 630MB/s to 1360MB/s, at the same time reducing the CPU utilization by a factor of ten.