Optimizations of primitive routines such as General Matrix Multiplication (GEMM) continue to advance the state of the art performance for their applications. Various libraries such as Basic Linear Algebra Subprograms (BLAS) exist that provide API interfaces to highly tuned, hardware-specific implementations. Applications such as deep learning push the limits of what is possible from these subroutines by necessitating unique optimizations like lowering numeric precision and data type bit-width, exploiting resiliency, and removing redundancy. Hardware plays a considerable role in the performance of a subroutine because of the inherent layout of memory and arithmetic structures such as the SIMD processors found in general CPU and GPU architectures. FPGAs play a unique role in this space because the reconfigurable circuits and routing provide a pipelined architecture capable of both SIMD and MIMD-like architectures. Within a pipeline, the capability of accelerating operations on FPGA through low-bit and fine-grained designs is typically only seen in ASICs. In this paper, we provide an overview of an OpenCL based GEMM accelerator design that exploits sparsity and compression to persist data in FPGA on-chip, fine-grained SRAM. The design includes support for successive GEMMs with added activation functions, making it suitable for some machine learning applications. Results are measured running on Intel's Arria 10 GX 1150 FPGA. Compared to non-sparse and non-persistent designs, we achieve a speedup towards the theoretical limit of 6.5× for our fine-grained implementation and 8× for our structured implementation. We measure over 1 TOP/s utilizing only 17% of Arria 10 DSP blocks for a 90% sparse design.

Convolutional neural networks (CNN) have been shown to maintain reasonable classification accuracy when quantized to lower precisions, however, quantizing to sub 8-bit activations and weights can result in classification accuracy falling below an acceptable threshold. Techniques exist for closing the accuracy gap of limited numeric precision networks typically by means of increasing computation. This results in a trade-off between throughput and accuracy and can be tailored for different networks through various combinations of activation and weight data widths. Customizable hardware architectures like FPGAs provide the opportunity for data width specific computation through unique logic configurations leading to highly optimized processing that is unattainable by full precision networks. Specifically, ternary and binary weighted networks offer an efficient method of inference for 2-bit and 1-bit data respectively. Most hardware architectures can take advantage of the memory storage and bandwidth savings that come along with a smaller datapath, but very few architectures can take full advantage of limited numeric precision at the computation level. In this paper, we present a hardware design for FPGAs that takes advantage of the bandwidth, memory, power, and computation savings of limited numerical precision data. We provide insights into the trade-offs between throughput and accuracy for various networks and how they map to our framework. Further, we show how limited numeric precision computation can be efficiently mapped onto FPGAs for both ternary and binary cases. Starting with Arria 10, we show a 2-bit activation and ternary weighted AlexNet running in hardware that achieves 3,700 images per second on the ImageNet dataset with a top-1 accuracy of 0.49. Using a hardware modeler designed for our low numeric precision framework we project performance most notably for a 55.5 TOPS Stratix 10 device running a modified ResNet-34 with only 3.7% accuracy degradation compared with single precision.

Convolutional neural networks have been shown to maintain reasonable classification accuracy when quantized down to 8-bits, however, quantizing to sub 8-bit activations and weights can result in classification accuracy falling below an acceptable threshold. Techniques exist for increasing accuracy of sub 8-bit networks typically by means of increasing computation resulting in a trade-off between throughput and accuracy and can be tailored for different networks through combinations of activation and weight precisions. Customizable hardware architectures like FPGAs provide opportunity for data width specific computation through unique logic configurations leading to highly optimized processing that is unattainable by full precision networks. Specifically, ternary and binary weighted networks offer an efficient method of inference for 2-bit and 1-bit data respectively. In this paper, we present a hardware design for FPGAs that takes advantage of the bandwidth, memory, and computation savings of limited numerical precision data. We provide insights into the trade-offs between throughput and accuracy for various networks and how they map to our framework. Further, we show how limited numeric precision computation can be efficiently mapped onto FPGAs for both ternary and binary cases. Starting with Arria 10, we show a 2-bit activation and ternary weighted AlexNet running in hardware that achieves 3,700 images per second on the ImageNet dataset with a top-1 accuracy of 0.49. Using a hardware modeler designed for our low numeric precision framework we project performance most notably for a 55.5 TOPS Stratix 10 device running a modified ResNet-34 with only 3.7% accuracy degradation compared with single precision.

FPGAs or ASICs? There is a long-running debate on this. FPGAs are extremely flexible while ASICs offer top efficiency but inflexible. We believe that FPGAs and ASICs are better together, to offer both flexible and efficient solutions. We propose single-package heterogeneous 2.5D integration of FPGAs and ASICs, using Intel's Embedded Multi-Die Interconnect Bridge (EMIB). Since the ASICs are separate chips from the FPGA, this approach (1) does not change FPGA fabric, allowing re-use of existing ecosystems (FPGA chips/packaging/boards/software/etc), and (2) allows freedom in the ASIC design (area/freq/process/etc unconstrained by FPGA fabric). This approach is more effective than developing traditional stand-alone ASICs. Intel® Stratix® 10 FPGAs already have EMIBs, enabling single-package integration with other chips, or "tiles". We propose leveraging them to mix-and-match any domain-specific ASICs with Stratix10 FPGAs.

FPGAs or ASICs? FPGAs are extremely flexible while ASICs offer top efficiency. We believe that FPGAs and ASICs are better together, to offer flexibility and efficiency. We propose single-package heterogeneous 2.5D integration of FPGAs and ASICs, using Intel's Embedded Multi-Die Interconnect Bridge (EMIB). Since the ASICs are separate chips from the FPGA, this approach (1) does not change FPGA fabric, allowing re-use of existing ecosystems (FPGA chips, packaging, boards, software), and (2) allows freedom in ASIC design (area/freq/process/etc unconstrained by FPGA fabric). Intel® Stratix® 10 FPGAs already have EMIBs, enabling single-package integration with other chips, or “tiles”. We propose leveraging them to mix-and-match any domain-specific ASICs with Stratix10 FPGAs. In particular, this work focuses on deep learning (DL) domain, which demands efficient tensor (matrix/vector) operations. We propose TensorTile ASICs for Stratix10 FPGAs to provide ASIC-level tensor performance, while relying on FPGA's flexibility for application-specific operations (e.g., Winograd). Our evaluation shows: (1) a small TensorTile offer much better tensor throughput than a large Stratix102800 FPGA; (2) FPGAs and TensorTiles mix-and-match provide scalable solutions (e.g., ~69 peak INT8 TOPs with 1×TensorTile+small Stratix10-400 FPGA, to ~194 peak FP16 TOPs with 6×TensorTiles+large Stratix10-2800); (3) AlexNet performance (performance/Watt) of Intel's DL FPGA design improved by 4× (3.3×) when enhanced with 2×TensorTiles.

Summary form only given. Binary weighted networks (BWN) for image classification reduce computation for convolutional neural networks (CNN) from multiply-adds to accumulates with little to no accuracy loss. Hardware architectures such as FPGA can take full advantage of BWN computations because of the irability to express weights represented as 0 and 1 efficiently through customizable logic. In this paper, we present an implementation on Intel®'s Xeon®processor with integrated FPGA to accelerate binary weighted networks. We interface Intel's Accelerator Abstraction Layer (AAL) with Caffe to provide a robust framework used for accelerating CNN. Utilizing the low latency Quick Path Interconnect (QPI) between the Broadwell Xeon®processor and Arria10 FPGA, we can perform fine-grained offloads for specific portions of the network. Due to convolution layers making up most of the computation in our experiments, we offload the feature and weight data to customized binary hardware in the FPGA for faster execution. An initial proof of concept design shows that by using both the Xeon processor and FPGA together we can improve the throughput by 2× on some layers and by 1.3× overall while utilizing only a small percentage of FPGA core logic.