Recently, training deep neural networks (DNNs) on edge devices has attracted much attention due to its strong adaptability and avoidance of private data transmission. However, limited computational, storage, and energy resources pose significant challenges for edge devices. The structural and computational redundancies in DNNs create opportunities for sparse training through model pruning and zero-computation skipping. Although feasible, the sparse training accelerator design encounters common issues, such as redundant data duplication and unbalanced workloads, caused by irregular sparsity. To address these issues, this paper proposes a sparse DNN training accelerator, SDTA, together with a hierarchical pre-fetching buffer and a dynamic scheduler to achieve high design efficiency. The SDTA is deployed on the FPGA XCVU3P platform. Compared to the prior FPGA-based accelerators and the GPU, SDTA improves the energy efficiency by up to 2.29×, the storage utilization efficiency by up to 7.37×, and the computational efficiency by up to 1.9×. Compared to the dense accelerator, it achieves a speedup of up to 5.88×, while ensuring model accuracy.

The transformer neural networks have achieved remarkable performance in both Natural Language Processing (NLP) and Computer Vision (CV) applications, with encoder-decoder architecture based on attention layers. However, implementing transformers on resource-constrained devices presents challenges due to the super-large network structures and nontrivial dataflows. Field-Programmable Gate Arrays (FPGA) have been a promising platform for Neural Network (NN) acceleration due to their design flexibility and customization. Existing FPGA-based implementations of transformers face efficiency and generality issues. This paper proposes HPTA, a high-performance accelerator for implementing transformers on FPGA. We analyze the structural features of transformer networks and design the accelerator with configurable processing element, optimized data selection and arrangement and efficient memory subsystem, to support various transformers. We evaluate the performance of HPTA with BERT and Swin Transformer, the typical transformer models in NLP and CV. HPTA achieves up to 44× and 29× inference time reductions compared with the CPU implementation, and up to 17× and 10x energy efficiency improvements compared with the GPU implementation, for BERT and Swin Transformer, respectively. Compared to the existing FPGA-based accelerators, HPTA shows performance improvements up to 1.3× and 1.8× in inference time compared to NPE and Vis-TOP, respectively.