Fast Fourier Transformation (FFT) has been widely recognized as an effective method for reducing the computational density of convolutional neural networks (CNNs). However, existing FFT-based CNN accelerators fail to fully exploit the algorithmic advantages of FFT while introducing significant additional hardware overhead. Consequently, this paper tries to figure out how to fully implement the benefit of FFT while saving the transformation overhead. First, we propose a CirConv-based multiplication reduction (CMR) algorithm, which leverages circular convolution and Hermitian Symmetry to eliminate unnecessary padding and optimize FFT scale selection. Building upon this algorithm, we further design a partially pipelined transformation-computation (PPTC) dataflow to reduce the parallelism and hardware complexity of FFT transformation. With the aid of the proposed CMR algorithm and PPTC dataflow, a high throughput FFT-based CNN accelerator is implemented on a Xilinx VCU118 FPGA platform, operating at 250 MHz. The experimental results demonstrate that, by combining Hermitian symmetry and PPTC dataflow, the LUT overhead for FFT/IFFT operations can be reduced to only 12.14% of the original. Furthermore, while deploying VGG16, the inference performance of our designed accelerator is 6213 GOPS, achieving a 2.17× to 29.31× improvement in actual performance and a 1.11× to 6.97× enhancement in DSP efficiency compared to the current works.

To overcome the trilemma between application flexibility, high efficiency, and minimal area/power overhead in baseband processors, this article proposes an area-efficient and reconfigurable accelerator for massive multiple-input-multiple-output (MIMO) systems, integrating a reconfigurable and heterogeneous processing element (PE) array featuring customized mixed-precision floating-point units (FPUs) to enable high-accuracy execution of diverse baseband operations (i.e., general matrix computation, inversion, mixed-radix fast fourier transform (FFT), decomposition, etc.) on matrices ranging from$4\times 4$–$32\times 32$. Addressing challenges posed by irregular partitioning and unbalanced decomposition in complex operations, an efficient heterogeneous mapping (EHeM) technology is introduced, achieving over 75% PE array utilization by minimizing external data access and reconfiguration overhead. Furthermore, a multibank memory structure with a dynamic access strategy (DAS) eliminates access conflicts and synchronizes memory-PE bandwidth across varying operators. This accelerator, validated on Xilinx Virtex-7 XC7VX690T, supports multiple baseband algorithms and signal processing operations, such as minimum mean square error (MMSE), FFT, and filtering. Evaluations show up to$75.41\times $throughput improvement and$239.30\times $efficiency improvement over operator-specific designs, and$1.10\times $–$6.02\times $higher efficiency than prior works for full MIMO processing.

Convolutional neural networks (CNNs) are evolving as they are applied to more diverse environments and more difficult challenges. The evolving induces various convolution modes (e.g., 1×1 convolution, 2-stride convolution and rectangle convolution) in current CNNs and makes it difficult for the hardware accelerators to efficiently support such various convolution modes. In this paper, it is found that an important difference of these convolution modes is the computation density. Therefore, the above convolution modes are regarded as structured sparse and claims that sparse-based design methodology can be applied for the implementation of the reconfigurable CNN accelerator. Subsequently, two critical architectural parameters, including input tile size and convolution engine (CE) scale, are evaluated based on Standard deviation of calculations (SDC), unsupported convolution mode (UCM) and unsuitable IFM size (UIS), DSP utilization ratio (DUR) as well as hardware resource overhead (HRO), respectively. With the aid of the optimal parameters, a high-parallelism and flexible CE array and a high-performance and reconfigurable CNN architecture are designed. The accelerator was implemented on a Xilinx VC709 FPGA and ran at a clock frequency of 300 MHz, achieving 921.60 to 1382.40 GOPS while supporting various convolution modes. Compared with previous dense-/sparse-based works, the proposed accelerator can realize 1.35× to 10.77× improvements on performance and 1.22× to 2.84× improvements on DSP efficiency while deploying VGG16.

CNN acceleration algorithms, including Winograd, Fast Fourier Transform (FFT) and Number Theoretic transform (NTT), have demonstrated their potential in efficiently operating current Convolutional Neural Networks (CNNs). However, deploying FFT algorithm for CNN acceleration would introduce significant invalid elements, unnecessary computations and unacceptable transformation overhead. To address these issues, this paper proposes a series of improved methods along with an FFT-based architecture for efficient and simplified CNN acceleration. First, a novel mixed-radix FFT algorithm is proposed for the reduction of invalid elements. Moreover, Hermitian symmetry is utilized to further reduce the scale of FFT transformation and the number of multiplications. Furthermore, an efficient FFT-based CNN accelerator with a resource-efficient transformation component and a multiplication-reduced PE array is designed. Our proposed accelerator is implemented based on Xilinx XCVU440 with a running frequency of 238MHz, achieving actual performance of 2109-2797 GOPS and DSP efficiency of 1.37-1.82 GOPS/DSP. Compared to previous works based on Winograd, FFT and NTT, our proposed accelerator can realize up to$9.42\times $speedup on actual performance and$1.11\times -6.41\times $speedup on DSP efficiency.

Leveraging sparsity in convolutional neural networks (CNNs) has emerged as a promising technique for enhancing the performance of CNN accelerators. However, despite achieving significant improvements in multiplier efficiency, the current sparse-based methods always fail to achieve a competitive performance and runtime latency compared with the conventional dense-based methods. This study posits that, because of the extremely irregular workload distribution in the input feature maps (IFMs), attaining a simultaneous improvement of the performance and multiplier efficiency in sparse-based accelerators is challenging. To address the above challenge, this study proposes a novel framework for eliminating irregularities at multi-level (i.e., dataflow-algorithm-hardware). Specifically, combined with filter decomposition and Winograd algorithms, a computation-oriented dataflow is designed for theoretical workload balancing under different convolution tasks. Furthermore, by exploiting the near-structured characteristic of IFMs, an online tile-based regularization scheme and a hybrid computation reduction method are designed to achieve a decreased and regular workload distribution. Finally, a large-scale sparse CNN accelerator, which integrates a row-merging scheme as well as a workload remapping method, is implemented to further eliminate hardware-level irregularities. The evaluation results show that our proposed methods can achieve 45.93%~76.59% multiplication savings when applied to VGG16, ResNet-34, and ResNet-50. Moreover, our accelerator can accomplish 3.06 TOPS and 2.86 sparsity extraction efficiency (SEE) while deploying VGG16, achieving a 1.17× to 3.45× enhancement on SEE compared with the state-of-the-art sparse-based accelerators.

Out-of-place constant-geometry (CG) NTT usually has a simple and uniform memory access pattern. However, out-of-place CG NTT always requires ping-pong memory, resulting in a memory capacity requirement of$2N$. Therefore, we propose a novel radix-4 in-place CG (IPCG) NTT/INTT that reduces the capacity requirement from$2N$to N. An area-efficient and dynamical reconfigurable polynomial multiplier (RAEPM) based on IPCG NTT is proposed to speed up polynomial multiplication over rings. In RAEPM, a Barrett modular multiplier using area-efficient radix-4 booth multiplier is designed to reduce area. In addition, an odd-bank buffer structure is proposed to achieve conflict-free memory mapping independent of polynomial length N and NTT/INTT stage. Moreover, we also proposed an efficient modular reduction for specific numbers and introduced a division equivalent method to eliminate the odd number modular reduction and odd number division in addressing. RAEPM is implemented on Xilinx VC709 FPGA and runs at 294MHz clock frequency. Compared with the prior pure NTT accelerators, under the same parameters, RAEPM achieves a decrease of 39.02%$\sim ~57.63$% in area-time complexity of equivalent LUT, and a decrease of 15.97%$\sim ~49.24$% in area-time complexity of equivalent FF. Compared with the prior NTT-based polynomial multipliers, under the same parameters, RAEPM achieves a decrease of 35.48%$\sim ~90.81$% in area-time complexity of equivalent LUT, and a decrease of 24.24%$\sim ~88.41$% in area-time complexity of equivalent FF.

Convolutional neural network (CNN) pruning is an effective way to reduce the computation requirement and improve the inference performance of standard convolutional layers. However, as for the low-computation-density layers in lightweight CNNs, pruning not only fails to improve their processing efficiency but also exacerbates the underutilization problem when deploying them on the convolutional engine. To efficiently execute these pruning-ineffective layers and further accelerate lightweight CNNs, a sparsity-adjustable CNN pruning method, which allows the pruning ratio to be adjusted, is proposed to prune the nonpruning-ineffective layers while shielding the pruning-ineffective layers. As a result, it achieves an additional 40% pruning ratio for nonpruning-ineffective layers with only 0.09% accuracy loss. Furthermore, a dense/sparse mixed-mode convolution computation scheme is designed to efficiently process the pruning- and nonpruning-ineffective layers using multiple acceleration techniques. Finally, a lightweight CNN accelerator is implemented on the Xilinx VCU118 FPGA platform. The comparison results with current studies present that this work can realize 1004.2 performance and 0.98 DSP efficiency while deploying MobileNetV2, achieving$1.26\times $-$6.13\times $enhancement on DSP efficiency.

General convolution acceleration, such as Winograd and FFT, is a promising direction to address the computational complexity of current convolutional neural networks (CNNs). However, the flexibility of these CNNs makes this kind of scheme always introduce massive redundant computations, damaging the acceleration effect. In this article, a two-stage splitting method for arbitrarily sized tensors and filters and a unified hardware architecture using layer-adaptive allocated Winograd units are proposed, achieving effective redundance elimination and unified architecture. First, a tensor adaptive presplitting method is proposed to divide the original tensors to match the rule of Winograd. Furthermore, a Winograd-based extended splitting scheme is designed to reduce the redundant calculations; therefore, a substantial reduction in multiplication operations in convolutional layers achieved 30.6%–75% savings. Finally, a unified hardware architecture with a layer-adaptive allocation method is proposed to evaluate and select the optimal Winograd F(${m}$,${r}$) units and input/output parallelisms. This architecture is evaluated based on the Xilinx XCVU9P platform and achieves 1.97/1.23/1.60/1.25 GOPS/DSP for AlexNet, VGG16, modified VGG16, and ResNet18, respectively. It achieves up to$5.81\times $improvements in DSP efficiency compared with previous FPGA-based designs.

Polynomial multiplication over rings is a significant bottleneck of ring learning with error (RLWE)-based encryption. To speed it up, the number theoretic transform (NTT) and Toom-Cook-4 (TC4) are commonly used algorithms. Compared with NTT, TC4 is less restrictive and more flexible. However, there is a large opportunity at the algorithm level to improve the Schoolbook algorithm and postprocessing of TC4. Therefore, we propose a novel and efficient Winograd-Schoolbook algorithm that reduces multiplication by 29.1% (N = 256). We also propose a fused and low-density postprocessing that simplifies the algorithm flow and reduces multiplication by 56.25%. In total, these two-part improvements reduce the multiplication of TC4 by 32.47%. A high throughput and efficiency TC4 polynomial multiplier (TCMW) is proposed to speed up polynomial multiplication over rings. In TCMW, a highly parallel full pipelined structure without data waiting between modules is designed to make the parallelism of each module match and avoid the storage of intermediate results. In addition, based on the improved TC4, data buffers with data reuse, elementwise vector multiplication (EWVM) arrays, and efficient interpolation arrays are all designed to improve the performance and efficiency of TCMW. Implemented on the Xilinx VC709 field programmable gate array (FPGA) platform, TCMW can perform a TC4-based$256\times 256$polynomial multiplication over rings with an unrestricted modulus (as long as its factors do not contain 3 or 5) every$1.89~\mu $s at a 385 MHz clock frequency. Compared with prior designs of TC4, under the same conditions, the throughput of TCMW achieves an improvement of$1.91\times \,\,\sim \,\,7.71\times $, and the efficiency of LUT and DSP achieve improvements of$1.31\times \,\,\sim \,\,3.67\times $and$1.87\times \,\,\sim \,\,4.92\times $, respectively.

Sparsification for convolutional neural networks (CNNs) and convolution acceleration algorithms such as the Winograd algorithm are two efficient ways to reduce the intensive computations of existing CNNs. To better combine the sparsification and Winograd algorithm, a close integration method is proposed to dynamically reduce the invalid parameters following the Winograd transformation. To address the limitation of data bandwidth, a hierarchical two-level storage structure and corresponding data scheduling scheme are proposed, which can realize a conflict-free scheduling process. In addition, an algorithm-hardware codesign method is proposed to efficiently and flexibly reduce the invalid computations led by the previous filter decomposition method. The accelerator is evaluated on Xilinx XCVU9P FPGA, reaching 412-MHz clock frequency. Compared to state-of-the-art designs, WRA-SS can achieve 1.54–$5.33\times $and 1.17–$7.39\times $performance improvement for VGG-16 under 80% weight sparsity and 0% weight sparsity, respectively.

To adapt to complex scenes and strict accuracy requirements, evolutions have unstoppably occurred in current convolutional neural networks (CNNs). However, these evolutions bring changes to filter size, convolution type, and sparsity, and such diversity leads to difficulties when adopting evolving CNNs in field-programmable gate array (FPGA)-based accelerators. This article proposes a dense-/sparse-aware CNN accelerator to achieve high PE utilization and configurability. First, a filter-based decomposition and clustering algorithm (FDCA) is proposed to change the various-sized filters into unified size filters. In addition, a sparse-aware filter transformation scheme (SFTS) is presented to dynamically eliminate invalid weights for sparse filters and accelerate dense filters. Based on the elimination of sparsity dependency, a hardware accelerator with a data–index decoupling workflow and an input channel schedule-distribution system is designed to take advantage of FDCA and SFTS. The proposed accelerator is implemented on a Xilinx ZCU102 platform at 300 MHz. With different CNN configurations, the digital signal processor (DSP) efficiencies for dense and unstructured sparse AlexNet and dense and structured sparse MobileNetV2 are 0.987, 2.025, 0.547, and 1.278 GOPS/DSP, respectively. Compared with previous dense- and sparse-based designs, the accelerator achieves up to a$4.263\times $speedup in DSP efficiency.

Mixed-radix fast Fourier transform (FFT) algorithms are widely adopted in high-performance communication systems, such as 5G systems. However, in the decimation-in-time (DIT) FFT algorithm, neither the input time-domain series nor the intermediate-stage data are fetched in natural order, which causes a mismatch between the processing element (PE) computation speed and the memory bandwidth. In addition, multiple duplicate interstage twiddle factor (TF) generation units are designed to provide interstage TFs in parallel, and such duplication results in considerable waste of computing and storage resources. In this article, we propose a flexible and reconfigurable architecture based on a mixed-radix FFT approach, supporting 61 different FFT sizes modes from 12 to 3240 points ($2^{\alpha }$or$12\times n$,$n \le270$, and$n$=$2^{\alpha } 3^{\beta } 5^{\gamma }$). A DIT-FFT-based multiple parallel changeable-radix butterfly unit (BU) is designed to improve hardware resource utilization. Around the PE array, a conflict-free access structure with a hardware-friendly address generating method is presented. In addition, a TF sharing and compression structure is designed to reduce the area and delay of the TF generation units. The FFT architecture is implemented in semiconductor manufacturing international corporation (SMIC) 40-nm CMOS technology with a working frequency of 483 MHz, performing at 654 MS/s for 2048-point and 1536-point FFTs. Compared to the state-of-the-art mixed-radix FFT designs, our architecture achieves improvements of up to$5.89\times $in throughput and area efficiency and supports more FFT modes.