In this paper we will explore the design and implementation of a multi-processor array consisting of a large number of soft parallel processors. We start with a new 16-SP 32-bit SIMT architecture that can run at 950MHz in Altera Agilex-7 devices. In one example, we create a 30 processor cluster that closes timing at over 930 MHz, and consumes only about 30% of a mid-range FPGA. This work is intended to enable the design of soft systems - where software and hardware approaches can be combined in an FPGA context.

Recent advances in soft GPGPU architectures have shown that a small (<10K LUT), high performance (770 MHz) processor is possible in modern FPGAs. In this paper we architect and evaluate soft SIMT processor banked memories, which can support high bandwidth (up to 16 ports) while maintaining high speed (over 770 MHz). We compare 9 different memory architectures, including simpler multi-port memories, and run a total of 51 benchmarks (different combinations of algorithms, data sizes and processor memories) to develop a comprehensive set of data which will guide the reader in making an informed memory architecture decision for their application. Our bench-marks are comprised of matrix transpositions (memory intensive) and FFTs (split between memory accesses, floating point, and integer computations) to provide a balanced evaluation. We show that the simpler (but more memory block intensive) multi-port memories offer higher performance than the more architecturally complex banked memories for many applications, especially for smaller memories, but the effective footprint cost of the multi-port memories quickly becomes prohibitive as dataset sizes increase. Our banked memory implementation results - high bandwidth, high Fmax, and high density - can be used for other FPGA applications as well, such as HLS (High Level Synthesis).

FPGAs offer a powerful and flexible platform to build complex systems on. But the potential - both in density and clock frequency - is often missed. In this work we present a massively parallel FFT, which can process a 32 K point FFT with 32-parallel IEEE-754 single-precision floatingpoint streams of complex data. Each core uses 1281 DSP Blocks, and can be packed into a near 100 % DSP Block density in an Altera Agilex-7 FPGA. Six such cores have been instantiated in a system, using 90 % of the device capability. In all cases our designs close timing at nearly 770 MHz, which is the restricted frequency of the DSP Block in floatingpoint mode. The contributions of this work are manifold. We demonstrate approaches that can be used to fill even larger FPGAs to high density with high performance. We also introduce optimizations for the FFT that can be used by all types of (including non-FFT) applications, from simple implementations to the very large parallel examples we use to demonstrate our research.

Multipliers have always been an important component of computer architecture, but the increasing relevance of Artificial Intelligence (AI) has brought about a massive increase in the number of multipliers on all compute platforms. At the same time, multiplier use for signal processing has also increased unabated. Multiplier architectures have not changed appreciably over the recent past. In this paper, we introduce a new technique for calculating partial products, which can be used with known compression tree and adder combinations. We demonstrate the efficiency of our new multiplier by reporting results from 800MHz to 2GHz in a current 7nm production library, and comparing to the well-known modified Booth’s radix 4 and radix 8 architectures.

Current soft processor architectures for FPGAs do not utilize the potential of the massive parallelism available. FPGAs now support many thousands of embedded floating point operators, and have similar computational densities to GPGPUs. Several soft GPGPU or SIMT processors have been published, but the reported large areas and modest Fmax makes their widespread use unlikely for commercial designs. In this paper we take an alternative approach, building the soft GPU microarchitecture around the FPGA resource mix available. We demonstrate a statically scalable soft GPGPU processor (where both parameters and feature set can be determined at configuration time) that always closes timing at the peak speed of the slowest embedded component in the FPGA (DSP or hard memory), with a completely unconstrained compile into a current Intel Agilex FPGA. We also show dynamic scalability, where a subset of the thread space can be specified on an instruction-by-instruction basis. For one example core type, we show a logic range -- depending on the configuration -- of 4k to 10k ALMs, along with 24 to 32 DSP Blocks, and 50 to 250 M20K memories. All of these instances close timing at 771 MHz, a performance level limited only by the DSP Blocks. We describe our methodology for reliably achieving this clock rate by matching the processor pipeline structure to the physical structure of the FPGA fabric. We also benchmark several algorithms across a range of data sizes, and compare to a commercial soft RISC processor.

Modular multiplication is a key kernel in many computing fields. What makes this function so challenging are the very large word sizes – sometimes in the thousands of bits – that are typically required for the target applications. In this paper we propose a modular multiplication implementation based on a multi-stage hybrid reduction technique. Our proposed approach uses a parameterized number of multiplier-based reduction stages followed by a memory-based reduction. This construction allows for the multiplier-based stages to take advantage of Karatsuba multiplication, resulting in a reduced number of DSP Blocks. Our method also allows specifying the number of multiplier-based stages which adjusts the ratio of multipliers to memory blocks. The resource utilization of the proposed architecture outperforms the existing state-of-the-art modular multiplication designs while offering a user-defined way of distributing resources between memory and DSP Blocks.

tThe CSAIL2019 time-lock puzzle is an unsolved cryptographic challenge introduced by Ron Rivest in 2019, replacing the solved LCS35 puzzle. Solving these types of puzzles requires large amounts of intrinsically sequential computations, with each iteration performing a very large (3,072-bit for CSAIL2019) modular multiplication operation. The complexity of each iteration is several times greater than known field-programmable gate array (FPGA) implementations, and the number of iterations has been increased by about 1,000x compared with LCS35. Because of the high complexity of this new puzzle, a number of intermediate, or milestone, versions of the puzzle have been specified. In this article, we present several FPGA architectures for the CSAIL2019 solver, which we implement on a medium-sized Intel Agilex device. We develop a new multi-cycle modular multiplication method, which is flexible and can fit on a wide variety of sizes of current FPGAs. We introduce a class of multi-cycle squarer-based architectures that allow for better resource and area trade-offs. We also demonstrate a new approach for improving the fitting and timing closure of large, chip-filling arithmetic designs. We used the solver to compute the first 23 out of 28 milestone solutions of the puzzle, which are the first reported results for this problem.

This work presents a set of techniques that allow implementing low-precision floating-point adders based on the embedded hard FP DSP Blocks available in contemporary Intel FPGAs. The presented architectures exploit the properties of these formats during exponent handling to obtain efficient implementations in terms of logic utilization. For instance, a half-precision floating-point adder implementation only requires 1 DSP Block and no extra logic. The newly available IEEE-754 compliant implementations can then be used as drop-in replacements in designs making use of these exact floating-point adder blocks. We present the case of a floating-point FFT implementation that benefits from these proposed architectures in order to substantially reduce logic utilization at the expense of using more DSP blocks.

The CSAIL2019 time-lock puzzle is an unsolved cryptographic challenge introduced by Ron Rivest in 2019, replacing the solved LCS35 puzzle. Solving these types of puzzles requires large amounts of intrinsically sequential computations (i.e. computations which cannot be parallelized), with each iteration performing a very large (3072-bit in the case of CSAIL2019) modular multiplication operation. The complexity of each iteration is several times greater than known FPGA implementations, and the number of iterations has been increased by about 1000x compared to LCS35. Because of the high complexity of this new puzzle, a number of intermediate, or milestone versions of the puzzle have been specified.

This paper introduces the eGPU, a SIMT soft processor designed for FPGAs. Soft processors typically achieve modest operating frequencies, a fraction of the headline performance claimed by modern FPGA families, and obtain correspondingly modest performance results. We propose a GPGPU architecture structured specifically to take advantage of both the soft logic and embedded features of the FPGA. We also consider the physical location of the embedded memories and DSP Blocks relative to the location and number of soft logic elements in order to have a design with balanced resources. Our goal is to create a high performance soft processor able to implement complex portions of FPGA system designs, such as the linear solvers commonly used in wireless systems, through push-button compilation from software. The eGPU architecture is a streaming multiprocessor (SM) machine with 512 threads. Each SM contains 16 scalar processors (SP). Both IEEE754 FP32 and INT32 integer arithmetic are supported. We demonstrate a single SM eGPU in an Intel Agilex device, requiring 5600 ALMs and 24 DSP Blocks, which closes timing at over 770 MHz from a completely unconstrained compile. Multiple eGPUs can also be tightly packed together into a single Agilex FPGA logic region, with minimal speed penalty.

Prior quantization methods focus on producing networks for fast and lightweight inference. However, the cost of unquantised training is overlooked, despite requiring significantly more time and energy than inference. We present a method for quantizing convolutional neural networks for efficient training. Quantizing gradients is challenging because it requires higher granularity and their values span a wider range than the weight and feature maps. We propose an extension of the Channel-wise Block Floating Point format that allows for quick gradient computation, using a minimal amount of quantization time. This is achieved through sharing an exponent across both depth and batch dimensions in order to quantize tensors once and reuse them during backpropagation. We test our method using standard models such as AlexNet, VGG, and ResNet, on the CIFAR10, SVHN and ImageNet datasets. We show no loss of accuracy when quantizing AlexNet weights, activations and gradients to only 4 bits training ImageNet.

Modular exponentiation, especially for very large integers of hundreds or thousands of bits, is a commonly used function in popular cryptosystems such as RSA. The complexity of this algorithm is partly driven by the very large word sizes, which require many - often millions - of primitive operations in a CPU implementation, or a large amount of logic when accelerated by an ASIC. FPGAs, with their many embedded DSP resources have started to be used as well. In almost all cases, the calculations have required multiple - occasionally many - clock cycles to complete. Recently, blockchain algorithms have required very low-latency implementations of modular multiplications, motivating new implementations and approaches.In this paper we show nine different high performance modular exponentiation for 1024-bit operands, using a 1024-bit modular multiplication as it’s core. Rather than just showing a number of completed designs, our paper shows the evolution of architectures which lead to different resource mix options. This will allow the reader to apply the examples to different FPGA targets which may have differing ratios of logic, memory, and embedded DSP blocks. In one design, we show a 1024b modular multiplier requiring 83K ALMs and 2372 DSPs, with a delay of 21.21ns.

The advent of AI has driven the exploration of high-density low-precision arithmetic on FPGAs. This has resulted in new methods in mapping both arithmetic functions as well as dataflows onto the fabric, as well as some changes to the embedded DSP Blocks. Technologies outside of the FPGA realm have also evolved, such as the addition of tensor structures for GPUs, as well as the introduction of numerous AI ASSPs, all of which have a higher claimed performance and efficiency than current FPGAs. In this article, we will introduce the Stratix 10 NX device, which is a variant of FPGA specifically optimized for the AI application space. In addition to the computational capabilities of the standard programmable soft-logic fabric, a new type of DSP Block provides the dense arrays of low-precision multipliers typically used in AI implementations. The architecture of the block is tuned for the common matrix-matrix or vector-matrix multiplications in AI, with capabilities designed to work efficiently for both small and large matrix sizes. The base precisions are INT8 and INT4, along with shared exponent support to support block FP16 and block FP12 numerics. All additions/accumulations can be done in INT32 or IEEE-754 single precision floating point (FP32), and multiple blocks can be cascaded together to support larger matrices. We will also describe methods by which the smaller precision multipliers can be aggregated to create larger multipliers that are more applicable to standard signal processing requirements. In the AI market, the FPGA must compete directly with other types of devices, rather than occupy a unique niche. Deterministic system performance is as important as the performance of individual FPGA elements, such as logic, memory, and DSP. We will show that the feed forward datapath structures that are needed to support the typical AI matrix-vector and matrix-matrix multiplication operations can consistently close timing at over 500 MHz on a mid-speed grade device, even if all of the Tensor Blocks on the device are used. We will also show a full-chip NPU processor implementation that out performs GPUs at the same process node for a variety of AI inferencing workloads, even though it has a lower operating frequency of 365 MHz. In terms of overall compute throughput, Stratix 10 NX is specified at 143 INT8/FP16 TOPs/FLOPs or 286 INT4/FP12 TOPS/FLOPs. Depending on the configuration, power efficiency is in the range of 1–4 TOPs or TFLOPs/W.

The advent of AI has driven the adoption of high density low precision arithmetic on FPGAs. This has resulted in new methods in mapping both arithmetic functions as well as dataflows onto the fabric, as well as some changes to the embedded DSP Blocks. Technologies outside of the FPGA realm have also evolved, such as the addition of tensor structures for GPUs, and also the introduction of numerous AI ASSPs, all of which have a higher claimed performance and efficiency than current FPGAs. In this paper we will introduce the Stratix 10 NX device (NX), which is a variant of FPGA specifically optimized for the AI application space. In addition to the computational capabilities of the standard programmable soft logic fabric, a new type of DSP Block provides the dense arrays of low precision multipliers typically used in AI implementations. The architecture of the block is tuned for the common matrix-matrix or vector-matrix multiplications in AI, with capabilities designed to work efficiently for both small and large matrix sizes. The base precisions are INT8 and INT4, along with shared exponent support for support block floating point FP16 and FP12 numerics. All additions/accumulations can be done in INT32 or IEEE754 single precision floating point (FP32), and multiple blocks can be cascaded together to support larger matrices. We will also describe methods by which the smaller precision multipliers can be aggregated to create larger multiplier that are more applicable to standard signal processing requirements. In terms of overall compute throughput, Stratix 10 NX achieves 143 INT8/FP16 TOPs/FLOPs, or 286 INT4/FP12 TOPS/FLOPs at 600MHz. Depending on the configuration, power efficiency is in the range of 1-4 TOPs or TFLOPs/W.

Encryption - especially the key exchange algorithms such as RSA - is an increasing use-model for FPGAs, driven by the adoption of the FPGA as a SmartNIC in the datacenter. While bulk encryption such as AES maps well to generic FPGA features, the very large multipliers required for RSA are a much more difficult problem. Although FPGAs contain thousands of small integer multipliers in DSP Blocks, aggregating them into very large multipliers is very challenging because of the large amount of soft logic required - especially in the form of long adders, and the high embedded multiplier count. In this paper, we describe a large multiplier architecture that operates in a multi-cycle format and which has a linear area/throughput ratio. We show results for a 2048-bit multiplier that has a latency of 118 cycles, inputs data every 9th cycle and closes timing at 377MHz in an Intel Arria 10 FPGA, and over 400MHz in a Stratix 10. The proposed multiplier uses 1/9 of the DSP resources typically used in a 2048-bit Karatsuba implementation, showing a perfectly linear throughput to DSP-count ratio. Our proposed solution outperforms recently reported results, in either arithmetic complexity - by making use of the Karatsuba techniques, or in scheduling efficiency - embedded DSP resources are fully utilized.

Barrett's algorithm is the most commonly known method of performing a modular multiplication, which is the core of many modern encryption algorithms such as RSA. Barrett's algorithm requires an accurate quotient estimation which in turn requires accurate multiplications. These multiplications operating on word sizes of thousands of bits are particularly expensive to implement in FPGAs, requiring many hundreds or even thousands of embedded DSP components along with large amounts of logic and routing. In this work we show that approximate quotient estimates as results of aggressive multiplier truncations can significantly reduce implementation cost. The looser modified Barrett's output [0; YM) is reduced to [0; M) using a shallow reduction technique based on table lookups and wide additions, taking advantage of new techniques which have recently been introduced for FPGA. We first use these techniques to develop an improved standard Barrett's implementation for 1024b modular multiplication, followed by our approximate method which reduces logic cost in the LSB truncated multiplier by approximately 10%. The effect is more pronounced for very large word sizes, where our relaxed error bounds in the LSB truncated multiplication can reduce the number of operations by 20%.

The importance of AI to FPGA has resulted in ever increasing low precision hard arithmetic features in newer devices. Many FPGAs, including those from Achronix, Intel, and Xilinx, have significantly increased the density of INT8 and INT9 embedded multipliers. Mainstream devices with these enhanced densities still support the traditional intermediate integer (typically 18-bit) multipliers, with IEEE-754 floating-point now becoming more prevalent as well.Recently, Intel introduced the Stratix 10 NX FPGA, which is targeted specifically at AI acceleration. This device contains a new type of AI-specific DSP Block with approximately an order of magnitude higher INT8 density than previous FPGA industry DSP Blocks. Larger standard FPGA integer precisions, however, are not directly supported. Intel has described some methods of aggregating larger multipliers from the NX Blocks, but these are somewhat smaller than typically used by DSP applications. Larger multiplications can also be useful for other AI applications, such as found in training. In this paper, we introduce the concept of signed tuples, which can be used to assemble signed multipliers into more useful larger precision multipliers by leveraging FPGA soft-logic inexpensively. We demonstrate several constructions of INT16 multipliers, with some modes requiring less than 3 ALMs per INT16 multiplier when implemented in a tensor format. We also describe the application of these methods to even larger multipliers and alternate constructs such as complex multiplication. We show that there is essentially no performance degradation or system fitting impact from our method. The mid-size NX device can support up 33 TOPs INT16 (from 29,700 constructed INT16 multipliers on a mid-speed grade device) with this approach, which is higher than any other current or announced monolithic die FPGA. Our methods are not limited to FPGA, or any particular starting precision, and so may be used for other aggregations as well.

”Soft” GPUs are overlays that implement GPGPU-like data parallel processor architectures in FPGA logic to make FPGAs as software-programmable as ”hard” GPGPUs. Unlike hard GPUs, soft GPU architectures can be specialized to further improve efficiency by leveraging FPGA’s flexibility. Prior work has shown the software programmability potential for soft GPUs but only studied general-purpose soft GPUs with minor specializations (e.g., FPGU, FlexGrip, MIAOW, and SCRATCH) or only domain-optimized for a particular application domain (e.g., PDL-FGPU for the persistent deep learning domain.) This paper proposes a soft GPU development framework to automate the creation of soft GPU instances with aggressive application-domain optimizations (i.e., domain-optimized GPUs, or DOGPUs) that consists of a baseline general soft GPU architecture ”template” with an improved architecture over prior general purpose soft GPUs, along with a customizable partition that enables a custom datapath (macro unit) to be inserted to optimize for a target application domain. Unlike the prior PDL-FGPU which targets the persistent deep learning domain, the proposed framework can be used to target optimization for any application domain. Our evaluation on a set of data parallel workloads shows that (i) the proposed general soft GPU architecture offers average speedup of 1.8x versus the best prior soft GPUs we know of (i.e., FGPU, PDL-FGPU), (ii) DO-GPUs with domain-optimizations provide an average of 218x speedup over general soft GPUs, (iii) the proposed framework enabled building six new domain-optimized soft GPU instances in a matter of days, and (iv) enables quick GPU-like development effort (hours), where code is concise (low 100s of lines) and can be compiled in seconds without FPGA EDA tools in the loop, assuming an appropriate soft DO-GPU bitstream for the application domain is already built.

Overlay architectures are a good way to enable fast development and debug on FPGAs at the expense of potentially limited performance compared to fully customized FPGA designs. When used in concert with hand-tuned FPGA solutions, performant overlay architectures can improve time-to-solution and thus overall productivity of FPGA solutions. This work tunes and specializes FGPU, an open source OpenCL-programmable GPU overlay for FPGAs. We demonstrate that our persistent deep learning (PDL )-FGPU architecture maintains the ease-of-programming and generality of GPU programming while achieving high performance from specialization for the persistent deep learning domain. We also propose an easy method to specialize for other domains. PDL-FGPU includes new instructions, along with micro-architecture and compiler enhancements. We evaluate both the FGPU baseline and the proposed PDL-FGPU on a modern high-end Intel Stratix 10 2800 FPGA in simulation running persistent DL applications (RNN, GRU, LSTM), and non-DL applications to demonstrate generality. PDL-FGPU requires 1.4–3× more ALMs, 4.4–6.4× more M20ks, and 1–9.5× more DSPs than baseline, but improves performance by 56–693× for PDL applications with an average 23.1% degradation on non-PDL applications. We integrated the PDL-FGPU overlay into Intel OPAE to measure real-world performance/power and demonstrate that PDL-FGPU is only 4.0–10.4× slower than the Nvidia V100.

Arithmetic is a key component and is ubiquitous in today’s digital world, ranging from embedded to high-performance computing systems. With machine learning at the fore in a wide range of application domains from wearables to automotive to avionics to weather prediction, sufficiently accurate yet low-cost arithmetic is the need for the day. Recently, there have been several advances in the domain of computer arithmetic, which includes high-precision anchored numbers from ARM, posit arithmetic, bfloat16, etc. as an alternative to IEEE 754-2008 compliant arithmetic. Optimizations on fixed-point and integer arithmetic are also being pursued actively for low-power computing architectures. Furthermore, approximate computing and transprecision/mixed-precision computing have been exciting areas of research forever. While academic research in the domain of computer arithmetic has a long history, industrial adoption of some of these new data types and techniques is in its early stages and expected to increase in the future. bfloat16 is an excellent example for this. In this paper, we bring academia and industry together to discuss the latest results and future directions for research in the domain of next-generation computer arithmetic, especially for edge computing.

Artificial Intelligence (AI) has become the fastest growing application area for FPGAs. Two types of numerics are needed. Training typically uses floating point arithmetic (which is now widely available as embedded functions in current FPGAs). Inference is typically calculated with lower precision integer numbers, which can be implemented with embedded functions, soft logic, or a combination of the two. INT8 performance is therefore used as a typical benchmarking metric for current FPGAs. Recent publications based on Xilinx devices show the extraction of two INT8 multipliers from a 24×18 multiplier. A paper from Intel describes how to obtain two INT8 multipliers from a 18×18 multiplier, with the help of a small amount of soft logic. In this paper we introduce a number of new INT8 multiplier techniques, starting with the Intel 18×18 multiplier approach. Using both memory and logic resources - for a more balanced use of the FPGA features - we improve the INT8 density, and also show a signed-magnitude (SM) 1.7 construct that is even smaller. To demonstrate the usability of these new multipliers, we develop a scalable systolic array, that contains up to 32,768 SM1.7 multipliers, or 28,800 INT8 multipliers, fit in an Intel Stratix 10 2800 device. Finally, we implement a system architecture that includes input and output flow buffering and control, which can be instantiated directly into a larger AI design, or can enable the FPGA to be used as a standalone accelerator. This system exceeds 400 MHz for the largest array on a mid-speed device (26 TOPS INT8), and can operate up to 600 MHz for smaller array sizes.

This paper describes architectural enhancements in Intel® Agilex™ FPGAs and SoCs. Agilex devices are built on Intel's 10nm process and feature next-generation programmable fabric, tightly coupled with a quad-core ARM processor subsystem, a secure device manager, IO and memory interfaces, and multiple companion transceiver tile choices. The Agilex fabric features multiple logic block enhancements that significantly improve propagation delays and integrate more effectively with the second-generation HyperFlexAgilex™ pipelined routing architecture. Routing connections are re-designed to be point-to-point, dropping intermediate connections featured in prior FPGA generations and replacing them with a wider variety of shorter wire types. Fine-grain programmable clock skew and time-borrowing were introduced throughout the fabric to augment the slack-balancing capabilities of HyperFlex registers. DSP capabilities are also extended to natively support new INT9/BFLOAT16/FP16 formats. Together, along with process and circuit enhancements, these changes support more than 40% performance improvement over the Stratix® 10 family of FPGAs.

With the advent of AI and machine learning as the highest profile FPGA applications, INT8 performance is currently one of the key benchmarking metrics. In current devices, INT8 multipliers must be extracted from higher precision multipliers. Recently, we reported the implementation of a mixed DSP Block and soft logic design, with 22,400 INT8 multipliers, and a system clock rate of 416MHz, on the Intel Stratix 10 2800 chip.

FPGAs are now being commonly used in the datacenter as smart Network Interface Cards (NICs), with cryptography as one of the strategic application areas. Public key cryptography algorithms in particular require arithmetic with thousands of bits of precision. Even an operation as simple as addition can be difficult for the FPGA when dealing with large integers, because of the high resource count and high latency needed to achieve usable performance levels with known methods. This paper examines the architecture and implementation of high-performance integer adders on FPGAs for widths ranging from 1024 to 8192 bits, in both single-instance and many-core chip-filling configurations. For chip-filling designs the routing impact of these wide busses are assessed, as they often have an impact outside the immediate locality of the structures. The architectures presented in this work show 1 to 2 orders magnitude reduction in the area-latency product over commonly used approaches. Routing congestion is managed, with near 100% logic efficiency (packing) for the adder function. Performance for these largely automatically placed designs are approximately the same as for carefully floor-planned non-arithmetic applications. In one example design, we show a 2048 bit adder in 5021 ALMs, with a latency of 6 clock cycles, at 628 MHz in a Stratix 10 E-2 device.

Interactive intelligent services, such as smart web search, are important datacenter workloads. They rely on dataintensive deep learning (DL) algorithms with strict latency constraints and thus require balancing both data movement and compute capabilities. As such, a persistent approach that keeps the entire DL model on-chip is becoming the new norm for realtime services to avoid the expensive off-chip memory accesses. This approach is adopted in Microsoft's Brainwave and is also provided by Nvidia's cuDNN libraries. This paper presents a comparative study of FPGA, GPU, and FPGA+ASIC in-package solutions for persistent DL. Unlike prior work, we offer a fair and direct comparison targeting common numerical precisions (FP32, INT8) and modern high-end FPGA (Intel® Stratix®10), GPU (Nvidia Volta), and ASIC (10 nm process), all using the persistent approach. We show that Stratix 10 FPGAs offer 2.7× (FP32) to 8.6× (INT8) lower latency than Volta GPUs across RNN, GRU, and LSTM workloads from DeepBench. The GPU can only utilize ~6% of its peak TOPS, while the FPGA with a more balanced on-chip memory and compute can achieve much higher utilization (~57%). We also study integrating an ASIC chiplet, TensorRAM, with an FPGA as system-in-package to enhance on-chip memory capacity and bandwidth, and provide compute throughput matching the required bandwidth. We show that a small 32 mm2 TensorRAM 10nm chiplet can offer 64 MB memory, 32 TB/s on-chiplet bandwidth, and 64 TOPS (INT8). A small Stratix 10 FPGA with a TensorRAM (INT8) offers 15.9× better latency than GPU (FP32) and 34× higher energy efficiency. It has 2× aggregate on-chip memory capacity compared to a large FPGA or GPU. Overall, our study shows that the FPGA is better than the GPU for persistent DL, and when integrated with an ASIC chiplet, it can offer a more compelling solution.

This paper will describe Fractal Synthesis, which is a new set of synthesis, clustering, and packing algorithms for FPGA devices, which dramatically increases the utilization and effective performance for arithmetic rich designs. The emergence of AI inferencing as a significant new FPGA application has brought some of the shortcomings of the FPGA and current design flows into focus. We describe new results where near 100% logic utilization of the FPGA is not only possible, but deterministic, with consistent high clock rates. Alternately, smaller datapaths can be synthesized, and combined to make chip filling designs. In one benchmark consisting of purely arithmetic datapath for a large Stratix®10 FPGA (E-2 speedgrade), we will show 92% logic utilization at 460 MHz for an automatically placed arithmetic datapath, and 410MHz with 97% logic utilization. Furthermore, we describe new results, where these performance and density level can be applied to non-arithmetic designs, by extending these techniques to placement.

Interactive intelligent services (e.g., smart web search) are becoming essential datacenter workloads. They rely on data-intensive artificial intelligence (AI) algorithms that do not use batch computation due to their tight latency constraints. Since off-chip data accesses have higher latency and energy consumption than on-chip accesses, a persistent AI approach with the entire model stored in on-chip memory is becoming the new norm for real-time AI. This approach is the cornerstone of Microsoft's Brainwave FPGA-based AI cloud and was recently added to Nvidia's cuDNN library. In this work, we implement, optimize and evaluate a Brainwave-like neural processing unit (NPU) on a large Stratix-10 FPGA. We benchmark it against a large Nvidia Volta GPU running cuDNN persistent AI kernels. Across real-time persistent RNN, GRU, and LSTM workloads, we show that Stratix-10 offers ~3× (FP32) and ~10× (INT8) better latency than GPU (FP32), which uses only ~6% of its peak throughput. Then, we propose TensorRAM, an ASIC chiplet for persistent AI that is 2.5D integrated with an FPGA in the same package. TensorRAM enhances the on-chip memory capacity and bandwidth, with enough multi-precision INT8/4/2/1 throughput to match that bandwidth. Multiple TensorRAMs can be integrated with Stratix-10. Our evaluation shows that a small 32-mm2 TensorRAM on 10nm offers 64MB of SRAMs with 32TB/s on-chiplet bandwidth and 64 TOP/s (INT8). A small Stratix-10 with a TensorRAM (INT8) offers 16× better latency and 34× energy efficiency compared to GPU (FP32). Overall, Stratix-10 with TensorRAM offers compelling and scalable persistent AI solutions.

With the advent of machine learning as perhaps the most high-profile application area for FPGAs, there is a compelling reason to improve the provision of smaller precision arithmetic on these devices. INT8 is commonly used for AI inferencing, and along with some additional soft logic for exponent handling, can be an effective solution for training as well. This paper describes techniques for efficiently extracting INT8 multipliers from commonly available INT18 multipliers found in many modern FPGAs. A small amount of soft logic - as little as 7 ALMs per INT8 multiplier - is required to provide pre or post multiplier correction to calculate two INT8 multiplies from a single 18x18 multiplier. We present two configurations for both signed and unsigned representations where two multiplications share one input operand. In addition to the individual INT8 variants, we present full device cases of 22,400 INT8 multipliers organized as DOT32 product arrays, with the soft logic tightly bound to the INT18 based DSP Blocks. A majority of the soft logic and routing in the device is left untouched, and available for application development.

Overlay architectures are a good way to enable fast development and debug on FPGAs at the expense of potentially limited performance when compared to fully customized FPGA designs. When used in concert with a hand-tuned FPGA solution, a performant overlay architecture can improve the time-to-solution and thus overall productivity of FPGA solutions. In this work, we tune and specialize FGPU, an open source OpenCL-programmable GPU overlay for FPGAs. We demonstrate that our PDL-FGPU architecture is able to maintain the ease-of-programming and generality of a software programmable soft GPU while achieving high performance due to specialization in the persistent deep learning domain. We also propose a easy method to specialize for different domains. PDL-FGPU includes new instructions, along with micro-architecture and compiler enhancements. We evaluate both the FGPU baseline and the proposed PDL-FGPU on a modern high-end Intel Stratix 10 2800 FPGA running a set of persistent DL applications (RNN, GRU, LSTM), as well as general non-DL applications to demonstrate generality. PDL-FGPU requires 1.5-3x more ALMs, 4.4-6.4x more M20ks, and 4.6-10x more DSPs than the FGPU baseline, but improves performance by 55-727x for persistent DL applications with an average 15% degradation on general non-PDL applications. We also demonstrate that the PDL-FGPU is only 4-7x slower than the Nvidia Volta V100 GPU.

Arithmetic based applications are one of the most common use cases for modern FPGAs. Currently, machine learning is emerging as the fastest growth area for FPG As, renewing an interest in low precision multiplication. There is now a new focus on multiplication in the soft fabric - very high-density systems, consisting of many thousands of operations, are the current norm. In this paper we introduce multiplier regularization, which restructures common multiplier algorithms into smaller, and more efficient architectures. The multiplier structure is parameterizable, and results are given for a continuous range of input sizes, although the algorithm is most efficient for small input precisions. The multiplier is particularly effective for typical machine learning inferencing uses, and the presented cores can be used for dot products required for these applications. Although the examples presented here are optimized for Intel Stratix 10 devices, the concept of regularized arithmetic structures are applicable to generic FPGA LUT architectures. Results are compared to Intel Megafunction IP as well as contrasted with normalized representations of recently published results for Xilinx devices. We report a 10% to 35% smaller area, and a more significant latency reduction, in the range of 25% to 50%, for typical inferencing use cases.

QR decomposition (QRD) is of increasing importance for many current applications, such as wireless and radar. Data dependencies in known algorithms and approaches, combined with the data access patterns used in many of these methods, restrict the achievable performance in software programmable targets. Some FPGA architectures now incorporate hard floating-point (HFP) resources, and in combination with distributed memories, as well as the flexibility of internal connectivity, can support high-performance matrix arithmetic. In this work, we present the mapping to parallel structures with inter-vector connectivity of a new QRD algorithm. Based on a Modified Gram-Schmidt (MGS) algorithm, this new algorithm has a different loop organization, but the dependent functional sequences are unchanged, so error analysis and numerical stability are unaffected. This work has a theoretical sustained-to-peak performance close to 100% for large matrices, which is roughly three times the functional density of the previously best known implementations. Mapped to an Intel Arria 10 device, we achieve 80us for a 256x256 single precision real matrix, for a 417 GFLOP equivalent. This corresponds to a 95% sustained to peak ratio, for the portion of the device used for this work.

Machine Learning is now one of the most active application areas for FPGAs. The more complex recurrent neural network (RNN) topologies require multiple non-linear activation functions, mainly tanh and sigmoid, per iteration. In this paper we will examine the impact of activation function quality - in both area and (especially) latency - on RNN performance. We present a number of architectures for these functions, for both half precision (IEEE754-2008 FP16) and single precision (IEEE754 FP32) floating-point representations. We describe how the IEEE754 single precision hard floating point (HFP) blocks available in current FPGAs ease the implementation of these functions, and we also give an alternate method the tanh function based on integer arithmetic. With the combination of exceptional internal memory bandwidth, direct support of high performance floating point dot products, and the new activation functions, we show that FPGAs can be a highly effective vehicle for these type of neural networks.

We present a new organization of the QR decomposition (QRD), which is optimized for implementation on parallel arithmetic structures, such as found in current FPGAs. Data dependencies are hidden in the pipeline depths of the datapaths, allowing implementations to approach 100% sustained to peak throughput. The algorithm presented here is based on the Modified Gram-Schmidt (MGS) method, and is designed for floating point (FP) arithmetic, with a combination of separate dot product and multiply-add datapaths. In this short paper, we concentrate on the description of the algorithm and architecture, rather than the implementation, of the QRD.

This paper presents an implementation of the floating-point (FP) tangent function, optimized for an FPGA containing hard floating point (HFP) DSP Blocks. This function inputs values in the interval [-π/2,π/2], uses the IEEE-754 single-precision (SP) format, and has an accuracy conforming to OpenCL requirements. The presented architecture is based on a combination of mathematical identities and properties of the tangent function in FP. The resultant design outperforms generic polynomial approximation methods targeting the same resource utilization spectrum, and provides better resource trade-offs than classical CORDIC-based implementations. The presented work is widely available as part of the Intel DSP Builder Advanced Blockset.

In this article we present a novel method for implementing floating point (FP) elementary functions using the new FP single precision addition and multiplication features of the Arria 10 and Stratix 10 DSP Block architecture. Our application examples are$\log (x)$and$\exp (x)$, two of the most commonly required functions for emerging datacenter and computing FPGA targets. We explain why the combination of new FPGA technology, and at the same time, a massive increase in computing performance requirement, fuels the need for this work. We show a comprehensive error analysis, and discuss various implementation trade-offs that demonstrate that the hard FP (HFP) Blocks, in conjunction with the traditional flexibility and connectivity of the FPGA, can provide a robust and high performance solution. The architectures presented in this work meet OpenCL accuracy requirements. Our methods map extensively to embedded structures, and therefore result in significant reduction in logic resources and routing stress compared to current methods. The methods allow leveraging the routing architectures introduced in the Stratix 10 device which results in high-function performance.

In this paper we will present a novel method for implementing floating point (FP) elementary functions using the new FP single precision addition and multiplication features of the Altera Arria~10 DSP Block architecture. Our application example will use log(x), one of the most commonly required functions for emerging datacenter and computing FPGA targets. We will explain why the combination of new FPGA technology, and at the same time, a massive increase in computing performance requirement, fuels the need for this work. We show a comprehensive error analysis, both for the overall function, and each subsection of the architecture, demonstrating that the hard FP (HFP) Blocks, in conjunction with the traditional flexibility and connectivity of the FPGA, can provide a robust and high performance solution. These methods create a highly accurate single precision IEEE754 function, which is OpenCL conformant. Our methods map directly to almost exclusively embedded structures, and therefore result in significant reduction in logic resources and routing stress compared to current methods, and demonstrate that newly introduced FPGA routing architectures can be leveraged to use almost no soft resources. We also show that the latency of the log(x) function can be changed independently of the architecture and function, allowing the performance of the function to be adjusted directly to the system clock rate.

This paper describes the architecture and implementation, from both the standpoint of target applications as well as circuit design, of an FPGA DSP Block that can efficiently support both fixed and single precision (SP) floating-point (FP) arithmetic. Most contemporary FPGAs embed DSP blocks that provide simple multiply-add-based fixed-point arithmetic cores. Current FP arithmetic FPGA solutions make use of these hardened DSP resources, together with embedded memory blocks and soft logic resources, however, larger systems cannot be efficiently implemented due to the routing and soft logic limitations on the devices, resulting in significant area, performance, and power consumption penalties compared to ASIC implementations. In this paper we analyse earlier proposed embedded FP implementations, and show why they are not suitable for a production FPGA. We contrast these against our solution -- a unified DSP Block -- where (a) the SP FP multiplier is overlaid on the fixed point constructs, (b) the SP FP Adder/Subtracter is integrated as a separate unit, and (c) the multiplier and adder can be combined in a way that is both arithmetically useful, but also efficient in terms of FPGA routing density and congestion. In addition, a novel way of seamlessly combining any number of DSP Blocks in a low latency structure will be introduced. We will show that this new approach allows a low cost, low power, and high density FP platform on current production 20nm FPGAs. We also describe a future enhancement of the DSP block that can support subnormal numbers.

This work describes the architecture of a new FPGA DSP block supporting both fixed and floating point arithmetic. Each DSP block can be configured to provide one single precision IEEE-754 floating multiplier and one IEEE-754 floating point adder, or when configured in fixed point mode, the block is completely backwards compatible with current FPGA DSP blocks. The DSP block operating frequency is similar in both modes, in the region of 500MHz, offering up to 2 GMACs fixed point and 1 GFLOPs performance per block. In floating point mode, support for multi-block vector modes are provided, where multiple blocks can be seamlessly assembled into any size real or complex dot products. By efficient reuse of the fixed point arithmetic modules, as well as the fixed point routing, the floating point features have only minimal power and area impact. We show how these blocks are implemented in a modern Arria 10 FPGA family, offering over 1 TFLOPs using only embedded structures, and how scaling to multiple TFLOPs densities is possible for planned devices.

Efficient elementary function implementations require primitives optimized for modern FPGAs. Fixed-point function generators are one such type of primitives. When built around piecewise polynomial approximations they make use of memory blocks and embedded multipliers, mapping well to contemporary FPGAs. Another type of primitive which can exploit the power series expansions of some elementary functions is floating-point polynomial evaluation. The high costs traditionally associated with floating-point arithmetic made this primitive unattractive for elementary function implementation on FPGAs. In this work we present a novel and efficient way of implementing floating-point polynomial evaluators on a restricted input range. We show on the atan(x) function in double precision that this very different technique reduces memory block count by up to 50% while only slightly increasing DSP count compared to the best implementation built around polynomial approximation fixed-point primitives.

This paper presents an FPGA-specific implementation of the floating-point tangent function. The implementation inputs values in the interval [-π/2,π/2], targets the IEEE-754 single-precision format and has an accuracy of 1 ulp. The proposed work is based on a combination of mathematical identities and properties of the tangent function in floating point. The architecture was designed having the {Stratix-IV} DSP and memory blocks in mind but should map well on any contemporary FPGA featuring embedded multiplier and memory blocks. It outperforms generic polynomial approximation targeting the same resource spectrum and provides better resources trade-offs than classical CORDIC-based implementations.The presented work is widely available as being part of the Altera DSP Builder Advanced Blockset.

Many applications require the evaluation of polynomials having floating-point coefficients - one example is rational polynomial approximation, often used to implement some special functions. The most resource efficient polynomial evaluation scheme (Horner) is costly to implement on FPGAs due to the high cost associated with floating-point arithmetic. Floating-point adders are particularly costly due to their alignment stages requiring large barrel shifters. In this work we present a novel FPGA-specific technique for evaluating polynomials using the Horner scheme. Our technique removes the majority of alignment shifters present in floating-point adders by building a fused evaluation operator. It pushes the possible alignment values of the monomials into tables containing multiple shifted coefficient instances which are selected using the exponent of the input argument. Compared to operator assembly this work reduces circuit latency by 30-50% and logic consumption by 40-60%. Our work can be easily extended to other polynomial evaluation methods.

User requirements for signal processing have increased in line with, or greater than, the increase in FPGA resources and capability. Many current signal processing algorithms require floating point, especially for military applications such as radar. Also, the increasing system complexity of these designs necessitate increased designer productivity, and floating point allows an easier implementation of the system model than the fixed point arithmetic that FPGA devices have been traditionally architected for. This article will review devices and methods for achieving consistent high performance system implementations in floating point. Single device designs at over 200 GFLOPs at the 40 nm node, and approaching 1 Teraflop at 28 nm will be described.

FPGA devices are now more capable at supporting floating point applications, but double precision IEEE754 systems are still very challenging to implement, with large amounts of logic, routing, and DSP resources needed. Double precision multiplier operators, requiring 54x54 multiplier cores, are now directly supported in the fabric, however, multiplier based algorithms for algebraic and elementary functions typically require larger multiplier functions, typically in excess of 60x60 bits precision. This paper will use the example of a double precision exp (ex) to explore tradeoffs with multiplier architectures mapped to current FPGAs devices. Different types of the larger multiplier architectures, including direct implementations and Karatsuba-Ofman algorithms will be compared, including their effect on the resource requirements and system performance of the exp function. Smaller multiplier architectures, which introduce a small maximum functional error, will also be examined.Alternate large multiplier architectures can provide the same accuracy with increased system performance, with up to 33% less multiplier resources. Smaller multiplier will reduce the multiplier resources by up to 50%, albeit with a small maximum error, in the range of 5 ulp.

This paper will describe the architecture of a compiler which will convert an untimed C description of a floating point expression into a synthesizable datapath optimized for FPGAs. The concept of floating point fused datapath synthesis will be reviewed, along with the expected functional efficiency gains. The dataflow graph structure used by the compiler will be detailed, followed by the description of the restructuring and optimizations, as well as the required data integrity considerations. In particular, datapath architecture considerations for improved FPGA fitting will be explored. Application examples for a matrix calculations will be used to illustrate the improvements of the compiled datapath compared to the traditional core based approach, and the mechanisms behind them.

In this paper we present an implementation of a Cholesky decomposition core, with IEEE754 single precision arithmetic. The datapaths are generated using fused datapath synthesis, created with an experimental floating point compiler tool, capable of fitting hundreds of floating point operators into a single device. We present a scalable architecture for both real and complex matrixes, on which we will report results for up to 128x128 real matrices. The concepts of fused datapath synthesis for FPGA floating point designs will be reviewed, and the application to the Cholesky algorithm detailed. Experimental results will be given to show that the accuracy of this method is superior to those expected from a traditional IEEE754 core based design flow.

Floating point arithmetic is used extensively in many applications across multiple market segments. While high performance IEEE754 floating point cores are available for FPGAs, a large datapath consisting of multiple cores is resource intensive, with often poor system performance. This paper will introduce a new approach to floating point datapath design for FPGAs, using fused datapath synthesis. The result is a more balanced, high performance implementation, typically saving 50% in both logic resources and latency. Using Stratix reg 3SE260 devices, 50 GFLOPs double precision and 125 GFLOPs single precision can be realized.