Three-dimensional convolutional neural networks (3D CNNs) have attracted great attention in many complex computer vision tasks. However, it is difficult to deploy 3D CNNs on practical applications due to high algorithmic complexity, imposing the urgent requirement for dedicated accelerators. In this paper, F3D, a fast algorithm for 3D CNNs, is proposed based on 3D Fast Fourier Transform (FFT) and achieves a significant algorithmic strength reduction. We then propose an F3D-based hardware architecture, featuring a flexible FFT module and an efficient partial sum aggregation module. Furthermore, a dataflow for efficient mapping of 3D CNNs is designed, leading to a significant reduction of memory access. To demonstrate the efficiency of the above-mentioned techniques, we implement the widely used 3D CNN model, C3D, as our benchmark on the Xilinx VC709 platform. The experimental result shows that compared with the state-of-the-art accelerator, our work achieves a considerable throughput up to 864.1 GOPs, along with 1.68x and 2.00 x efficiency improvement on energy and DSP utilization, respectively.

To achieve higher accuracy in machine learning tasks, very deep convolutional neural networks (CNNs) are designed recently. However, the large memory access of deep CNNs will lead to high power consumption. A variety of hardware-friendly compression methods have been proposed to reduce the data transfer bandwidth by exploiting the sparsity of feature maps. Most of them focus on designing a specialized encoding format to increase the compression ratio. Differently, we observe and exploit the sparsity distinction between activations in earlier and later layers to improve the compression ratio. We propose a novel hardware-friendly transform-based method named 1D-Discrete Cosine Transform on Channel dimension with Masks (DCT-CM), which intelligently combines DCT, masks, and a coding format to compress activations. The proposed algorithm achieves an average compression ratio of 2.9× (53% higher than the state- of-the-art transform-based feature map compression works) during inference on ResNet-50 with an 8-bit quantization scheme.

Designing hardware accelerators for convolutional neural networks (CNNs) has recently attracted tremendous attention. Plenty of existing accelerators are built for dense CNNs or structured sparse CNNs. By contrast, unstructured sparse CNNs can achieve higher compression ratio with equivalent accuracy. However, their corresponding hardware implementations generally suffer from load imbalance and conflict access to on-chip buffers, which results in under utilization of processing elements (PEs). To tackle these issues, we propose a hardware/power-efficient and highly flexible architecture to support both unstructured and structured sparse CNNs with various configurations. Firstly, we propose an efficient weight reordering algorithm to preprocess compressed weights and balance the workload of PEs. Secondly, an adaptive on-chip dataflow, namely hybrid parallel (HP) dataflow, is introduced to promote weight reuse. Thirdly, the partial fusion scheme, which was first introduced in one of our prior works, is incorporated as the off-chip dataflow. Benefited from dataflow optimizations, the repetitive data exchanges between on-chip buffers and external memories are significantly reduced. We implement the design on the Intel Arria10 SX660 platform and evaluate with MobileNet-v2, ResNet-50, and ResNet-18 on ImageNet dataset. Compared to existing sparse accelerators on FPGAs, the proposed accelerator can achieve 1.35 ~ 1.81× improvement in power efficiency with the same sparsity. Compared to prior dense accelerators, this accelerator can achieve an improvement of 1.92 ~ 5.84× in DSP efficiency.