In this article, we survey existing academic and commercial efforts to provide Field-Programmable Gate Array (FPGA) acceleration in datacenters and the cloud. The goal is a critical review of existing systems and a discussion of their evolution from single workstations with PCI-attached FPGAs in the early days of reconfigurable computing to the integration of FPGA farms in large-scale computing infrastructures. From the lessons learned, we discuss the future of FPGAs in datacenters and the cloud and assess the challenges likely to be encountered along the way. The article explores current architectures and discusses scalability and abstractions supported by operating systems, middleware, and virtualization. Hardware and software security becomes critical when infrastructure is shared among tenants with disparate backgrounds. We review the vulnerabilities of current systems and possible attack scenarios and discuss mitigation strategies, some of which impact FPGA architecture and technology. The viability of these architectures for popular applications is reviewed, with a particular focus on deep learning and scientific computing. This work draws from workshop discussions, panel sessions including the participation of experts in the reconfigurable computing field, and private discussions among these experts. These interactions have harmonized the terminology, taxonomy, and the important topics covered in this manuscript.

AIgean , pronounced like the sea, is an open framework to build and deploy machine learning (ML) algorithms on a heterogeneous cluster of devices (CPUs and FPGAs). We leverage two open source projects: Galapagos , for multi-FPGA deployment, and hls4ml , for generating ML kernels synthesizable using Vivado HLS. AIgean provides a full end-to-end multi-FPGA/CPU implementation of a neural network. The user supplies a high-level neural network description, and our tool flow is responsible for the synthesizing of the individual layers, partitioning layers across different nodes, as well as the bridging and routing required for these layers to communicate. If the user is an expert in a particular domain and would like to tinker with the implementation details of the neural network, we define a flexible implementation stack for ML that includes the layers of Algorithms, Cluster Deployment & Communication, and Hardware. This allows the user to modify specific layers of abstraction without having to worry about components outside of their area of expertise, highlighting the modularity of AIgean . We demonstrate the effectiveness of AIgean with two use cases: an autoencoder, and ResNet-50 running across 10 and 12 FPGAs. AIgean leverages the FPGA’s strength in low-latency computing, as our implementations target batch-1 implementations.

Machine learning (ML) in the past decade has been one of the most popular topics of research within the computing community. Interest within the computing field ranges across all levels of the computation stack. We show this stack in Figure 1. This work introduces an open framework, called AIgean, to build and deploy machine learning (ML) algorithms on a heterogeneous cluster of devices (CPUs and FPGAs). Users can flexibly modify any layer of the machine learning stack in Figure 1 to suit their need. This allows both machine learning domain experts to focus on higher algorithmic layers, and distributed systems experts to create the communication layers below.

Design verification is an important though time-consumingaspect of hardware design. A good testbench should supportperforming functional coverage of a design by making it easy to implement tests and determine which tests are being performed. However, for complex designs, creating and main-taining effective testbenches can take increasing amounts of time away from actual design. A further complication is there may be two development flows: conventional hardware written in a hardware description language (HDL) such as Verilog orVHDL and high-level synthesis (HLS). In the HLS approach, the hardware is specified in a higher-level language (HLL) and then converted to an HDL through HLS tools. In this flow, testbenches for the design are written in the same HLLand cosimulation is used to verify the generated HDL. Due totool restrictions, cosimulation may not always work. In VivadoHLS [1] for example, the design must contain control signals to define when to start and stop the module or the initiation interval for new data must be one cycle. Without cosimulation, the user must write an HDL testbench manually in addition to a testbench in the HLL for preliminary verification. To simplify writing testbenches, we present Sonar: an open-source Python library to write cross-language testbenches. From a common source script, Sonar can generate testbenches written in SystemVerilog (SV) and C++. These files can then be imported into standard simulation tools such as ModelSim[2] or Vivado HLS and run. The use of Python makes it easy to extend Sonar with higher layers of abstraction for testbenches and integrate it with other software platforms.Sonar is available at https://github.com/UofT-HPRC/sonar.

In this work we present a heterogeneous deployment stack, calledGalapagos, that includes the abstraction of individual nodes (FPGAsand CPUs), the communication protocols between nodes and theorchestration and connection of these nodes into clusters. The stackwe create is also highly modular, allowing users to explore a designspace in the implementation of their cluster such as different net-work protocols or communication layers. The communication layerwe have currently implemented within our hardware stack, calledHUMboldt, handles heterogeneous communication between multi-ple FPGAs and CPUs. We implementHUMboldtusing High-LevelSynthesis (HLS) to ensure functional portability of communicatingkernels, allowing us to prototype hardware kernels in software. Ourresults have shown that our modular approach to this heterogeneousdeployment stack has introduced very little area and latency over-head in the FPGAs and can still perform at line-rate, bottleneckedsolely by the network links connecting the nodes. Our results alsohighlight the scalability of our design as our performance remainslimited by the network links when the cluster size increases.

The economies of scale afforded by cloud computing has been a driving force behind the rapid development and deployment of new cloud-based network applications and services. With the massive growth of IoT devices, we expect a sharp rise in the volume of traffic seen going to and coming from cloud datacenters, which will continue to grow over the next several years. Network Function Virtualization (NFV) is a recent concept which promises to grant network operators the required flexibility to quickly develop and provision new network functions and services in the cloud. As NFV is agnostic to the computing resource, we foresee scenarios where unconventional resources such as FPGAs and GPUs will be of benefit. To this end, we present an architecture based on Software-Defined Infrastructure (SDI) which offers an abstracted control and management interface over virtualized heterogeneous resources in the cloud. Through a unified set of APIs, this architecture enables both application developers and network operators to dynamically deploy and manage new services in the cloud alongside the underlying network that interconnects them, all in a fully software-defined manner. We demonstrate and evaluate an implementation of our NFV-enablement architecture using the SAVI testbed, a multi-tier and SDN-enabled cloud containing virtualized heterogeneous compute resources.

We present a framework for creating heterogeneous virtualized network function (VNF) service chains from cloud data center resources. Traditionally, these functions are packaged in software images within a catalog of networking applications that can be loaded onto a virtual machine CPU, and can be offered to users as a service. Our framework combines the best of both software and hardware by allowing users to chain traditional software-based VNFs with hardware-based VNFs that the user provides as an IP to generate a bitstream or a pre-generated VNF as part of a library. To accomplish this, our framework first creates the hardware bitstreams and programs the FPGA VNFs, loads any software VNFs requested, and programs the network to daisy chain the VNFs together. Furthermore, this enables an incremental design flow where the user can start by implementing a chain of VNFs in software and incrementally substitute software VNFs for their hardware counterparts. Our paper investigates two case studies to show the ability to switch between hardware and software VNFs in our framework and to demonstrate the benefit of using hardware VNFs. The first study is signature matching at fixed offsets, similar to matching packet headers. In this case study, the CPU can keep up at line-rate using specialized networking drivers. The second case study involves string matching within a packet, which requires scanning through the entire frame. In this case, the CPU performance drops to approximately 20 percent of the input rate, whereas the FPGA can continue to keep up at line-rate.

We present a novel method of using cloud-based virtualized reconfigurable hardware to enhance the functionality of OpenFlow Software-Defined Networks. OpenFlow is a capable and popular SDN implementation, but when users require new or unsupported packet-processing, software processing in the OpenFlow controller cannot provide multi-gigabit rates. Our method sees packet flows redirected through virtualized hardware with custom-designed packet-processing engines that can add new capabilities to an OpenFlow network, while retaining line-rate processing. A case study shows this can be achieved with virtually no loss in throughput and minimal latency overheads.

We present an efficient multi-query event stream platform to support query processing over high-frequency event streams. Our platform is built over reconfigurable hardware -- FPGAs -- to achieve line-rate multi-query processing by exploiting unprecedented degrees of parallelism and potential for pipelining, only available through custom-built, application-specific and low-level logic design. Moreover, a multi-query event stream processing engine is at the core of a wide range of applications including real-time data analytics, algorithmic trading, targeted advertisement, and (complex) event processing.