A multitude of accelerators have been designed to accelerate the inference of widely-used Convolutional Neural Networks (CNNs). They can primarily be classified into two architectures: the Overlay architecture, which accelerates layer by layer, and the Dataflow architecture, which accelerates the entire model. Each has its own strengths and weaknesses, and we will propose a novel architecture to capitalize on their strengths and mitigate their weaknesses. Our architecture allows optimization for different target network structures. Internally, it features multiple PE Arrays, and during runtime, it can flexibly switch between different interconnection modes among these PE Arrays: the serial execution mode, accelerating an entire block at once, and the parallel execution mode, accelerating a single layer at a time. Its minimum area requirement is close to that of typical Overlay architecture accelerators, while its throughput per unit area far exceeds that of existing state-of-the-art accelerators. When executing the widely-used 8-bit quantized MobileNetV2, we observed an exceptionally high throughput of 2331 FPS (frames per second) on the mid-range FPGA ZU7EV, while requiring only 2.89 GiB/s of off-chip memory bandwidth. Running other networks also demonstrated high efficiency and reasonable off-chip memory bandwidth requirements.

In recent years, Convolutional Neural Networks (CNNs) have been widely used in many fields. Many accelerators are designed to run CNN inference on hardware. They are classified into two types: Overlay and Dataflow. In this paper, we propose a new architecture of CNN Inference accelerators to leverage the strengths of each type. It is flexible, fast, and has low off-chip memory bandwidth requirements. The accelerator based on our new architecture can be configured to fit the CNN structure and FPGA capacity. It basically runs CNNs by blocks, whereas it has a fallback path to run single Convolution or Fully Connected layers efficiently. Our architecture demonstrates significant advantages when running networks composed of inverted residual blocks. We archived 1954FPS (frames per second) when running 8-bit quantized MobileNetV2 on a mid-range FPGA ZU7EV, and the off-chip memory bandwidth requirement is as low as 2.47GiB/s. It also runs on cost-optimized FPGAs, and archived 505FPS on ZU3EG. The throughputs of ours are higher than the other types, while the minimum FPGA capacity and the off-chip memory bandwidth requirement of ours are reasonable.