Summary Apache Spark has become one of the most popular engines for big data processing. Spark provides a platform‐independent, high‐abstraction programming paradigm for large‐scale data processing by leveraging the Java framework. Though it provides software portability across various machines, Java also limits the performance of distributed environments, such as Spark. While it may be unrealistic to rewrite platforms like Spark in a faster language, a more viable approach to mitigate its poor performance is to accelerate the computations while still working within the Java‐based framework. This paper demonstrates the feasibility of incorporating Field‐Programmable Gate Array (FPGA) acceleration into Spark and presents the performance benefits and bottlenecks of our FPGA‐accelerated Spark environment using a MapReduce implementation of the k‐means clustering algorithm, to show that acceleration is possible even when using a hardware platform that is not well optimized for performance. An important feature of our approach is that the use of FPGAs is completely transparent to the user through the use of library functions, which is a common way by which users access functions provided by Spark. Power users can further develop other computations using high‐level synthesis.

In the field of deep learning, efficient computational hardware has come to the forefront of the large scale implementation and deployment of many applications. In the process of designing hardware, various characteristics of hardware platforms have been studied in order to best implement the high computational demand, high memory bandwidth, and flexibility of networks. In addition to design space exploration of kernels, kernel design must be seen in the context of full system architectures or in terms of the combination of deep learning and other types of applications whether video encoding/decoding or analytics, speech recognition, or the multitude of potential applications combining deep learning kernels with tightly integrated coprocessor architectures. Kernel sizes, on-chip and off-chip memories, numeric datatypes and efficient compute architectures all must be merged into optimal design choices for both performing computations with maximum efficiency as well as programmable flexibility.

Summary form only given. Apache Spark has become one of the most popular engines for big data processing. Spark provides a platform-independent, high-abstraction programming paradigm for large-scale data processing by leveraging the Java frame-work. Though it provides software portability across various machines, Java also limits the performance of distributed environments, such as Spark. While it may be unrealistic to rewrite platforms like Spark in a faster language, a more viable approach to mitigate its poor performance is to accelerate the computations while still working within the Java-based framework. This work demonstrates the feasibility of incorporating FPGA acceleration into Spark, and uses a MapReduce implementation of the k-means clustering algorithm to show that acceleration is possible even when using a hardware platform that is not well-optimized for performance. An important feature of our approach is that the use of FPGAs is completely transparent to the user through the use of library functions, which is a common way by which users access functions provided by Spark. Power users can further develop other computations using high-level synthesis.

Due to rapidly expanding data size, there is increasing need for scalable, high-performance, and low-energy frameworks for large- scale data computation. We build a dataflow architecture that harnesses FPGA resources within a distributed analytics platform creating a heterogeneous data analytics framework. This approach leverages the scalability of existing distributed processing environments and provides easy access to custom hardware accelerators for large-scale data analysis. We prototype our framework within the Apache Spark analytics tool running on a CPU-FPGA heterogeneous cluster. As a specific application case study, we have chosen the MapReduce paradigm to implement a multi-purpose, scalable, and customizable RTL accelerator inside the FPGA, capable of incorporating custom High-Level Synthesis (HLS) MapReduce kernels. We demonstrate how a typical MapReduce application can be simply adapted to our distributed framework while retaining the scalability of the Spark platform.