MoE (Mixture-of-Experts) mechanism has been widely adopted in transformer-based models to facilitate further expansion of model parameter size and enhance generalization capabilities. However, the practical deployment of MoE mechanism for transformer on resource-constrained platforms, such as FPGA, remains challenging due to heavy memory footprints and impractical runtime costs introduced by the MoE mechanism. Diving into the MoE mechanism, we raise two key observations: (1) Expert weights are heavy but cold, making it ideal to leverage expert weight sparsity. (2) There exists highly skewed expert activation paths for MoE layers in transformer-based models, making it feasible to conduct expert prediction and prefetching. Motivated by these two observations, we propose FLAME, the first algorithm-hardware co-optimized MoE accelerating framework designed to fully leverage MoE sparsity for efficient transformer deployment on FPGA. First, to leverage expert weight sparsity, we integrate an N:M pruning algorithm, allowing for the pruning of expert weights without significantly compromising model accuracy. Second, to settle expert activation sparsity, we propose a circular expert prediction (CEPR) strategy. CEPR prefetches expert weights from external storage to on-chip cache before the activated expert index is determined. Last, we co-optimize both MoE sparsity through the introduction of an efficient pruning-aware expert buffering (PA-BUF) mechanism. Experimental results demonstrate that FLAME achieves 84.4% accuracy of expert prediction with merely two expert caches on-chip. In comparison with CPU and GPU, FLAME achieves 4.12× and 1.49× speedup, respectively.

Transformer models have become popular in various AI applications due to their exceptional performance. However, their impressive performance comes with significant computing and memory costs, hindering efficient deployment of Transformer-based applications. Many solutions focus on leveraging sparsity in weight matrix and attention computation. However, previous studies fail to exploit unified sparse pattern to accelerate all three modules of Transformer (QKV generation, attention computation and FFN). In this paper, we propose FNM-Trans, an adaptable and efficient algorithm-hardware co-design aimed at optimizing all three modules of the Transformer by fully harnessing N : M sparsity. At the algorithm level, we fully explore the interplay of dynamic pruning with static pruning under high N : M sparsity. At the hardware level, we develop a dedicated hardware architecture featuring a custom computing engine and a softmax module, tailored to support varying levels of N : M sparsity. Experiment results show that, our algorithm optimizes accuracy by 11.03% under 2:16 attention sparsity and 4:16 weight sparsity, compared to other methods. Additionally, FNM-Trans achieves speedups of 27.13× and 21.24× over Intel i9-9900X and NVIDIA RTX 2080 Ti, respectively, and outpaces current FPGA-based Transformers by 1.88× to 36.51×.

Recent years have witnessed Transformers emerge as a groundbreaking innovation in the Natural Language Processing (NLP) field. Unlike Recurrent Neural Network (RNN) models, Transformers process sequences in parallel, boosting accuracy for longer sequences. However, Transformers face challenges with extended processing time. This is particularly due to the requirement of padding inputs to match the longest sentence in a batch, thereby increasing computational demands. In this paper, we present Fitop-Trans, the first algorithm-hardware co-optimized framework using Fixed-Length Token Pruning strategy while deploying Transformers on FPGA. At the algorithmic level, we propose Fixed-Length Token Pruning. It is a novel pruning method which can maximize hardware efficiency in attention computation, aimed at eliminating unimportant tokens before the first layer. On the hardware side, a token selector is designed for Fixed-Length Token Pruning, which minimizes off-chip memory traffic. In addition, a partitionable Systolic Array (SA) is adopted, which is capable of handling varying input lengths and maximizing Digital Signal Processor (DSP) resource utilization. Furthermore, a scheduling module is designed to optimize hardware resource allocation and enhance pipeline attention throughput. Experimental results reveal that our hardware design on FPGA achieves a speedup of $580 \times$ and $6.39 \times$ in latency compared to Intel Xeon Gold CPU and NVIDIA GeForce RTX 3090.

Existing accelerators for transformer networks with field-programmable gate array (FPGA) either focus only on attention computation or suffer from fixed data streams without flexibility. Moreover, compression and approximation methods of transformer networks have the potential for further optimization. In this article, we propose a low-latency FPGA-based overlay processor, named LTrans-OPU for general accelerations of transformer networks. Specifically, we design a domain-specific overlay architecture, including a computation unit for matrix multiplication of arbitrary dimensions. An instruction set customized for our overlay architecture is also introduced, dynamically controlling data flows by generated instructions. In addition, we introduce a hybrid pruning method common to various transformer networks, along with an efficient non-linear function approximation method. Experimental results show that our design is rather competitive and has low latency. LTrans-OPU achieves 11.10-32.20× speedup compared with CPU and 2.44-6.18 × latency reduction compared with GPU. We also observe 2.36-12.43 × lower latency compared with customized FPGA/ASIC accelerators, and can be 3.10× faster than NPE.

Transformer models have been widely adopted in the field of Natural Language Processing (NLP) and Computer Vision (CV). However, the excellent performance of Transformers comes at the cost of heavy memory footprints and gigantic computing complexity. To deploy Transformers on resource constrained platforms, e.g., FPGA, diverse weight pruning strategies have been proposed. However, pattern pruning, as an alternative pruning method, is not well explored in the context of Transformers. In this paper, we propose PP-Transformer, a framework specifically designed to efficiently deploy Transformer models on FPGA using pattern pruning. At the algorithm level, we leverage pattern pruning, a coarse-grained structured pruning strategy, to reduce parameter storage. Meanwhile, we have developed a dedicated hardware architecture, featuring a custom computing engine tailored to support pattern pruning algorithm. Experimental results demonstrate that our algorithm achieves up to$2.26\times$reduction in parameter storage with acceptable accuracy degradation. Additionally, our hardware implementation exhibits$839.72\times$and$5.72\times$speedup in comparison to CPU and GPU implementations.