Bit-serial architectures (BSAs) are becoming increasingly popular in low-power neural network processor (NNP) designs for edge scenarios. However, the performance and energy efficiency of state-of-the-art BSA NNPs heavily depends on both the proportion and distribution of ineffectual weight bits in neural networks (NNs). To boost the performance of typical BSA accelerators, we present Bit-Pruner, a software approach to learn BSA-favored NNs without resorting to hardware modifications. Bit-Pruner not only progressively prunes but also restructures the nonzero bits in weights so that the number of nonzero bits in the model can be reduced and the corresponding computing can be load-balanced to suit the target BSA accelerators. On top of Bit-Pruner, we further propose a Pareto frontier optimization algorithm to adjust the bit-pruning rate across network layers and fulfill diverse NN processing requirements in terms of performance and accuracy for various edge scenarios. However, an aggressive Bit-Pruner can lead to nontrivial accuracy loss, especially for lightweight NNs and complex tasks. To this end, the alternating direction method of multipliers (ADMMs) is adapted to the retraining phase in Bit-Pruner to smooth the abrupt disturbance due to bit-pruning and enhance the resulting model accuracy. According to the experiments, Bit-Pruner increases the bit-sparsity up to 94.4% with negligible accuracy degradation and achieves an optimized tradeoff between NN accuracy and energy efficiency even under very-aggressive performance constraints. When pruned models are deployed onto typical BSA accelerators, the average performance is$2.1\times $and$1.6\times $higher than the baseline networks without pruning and those with classical weight pruning, respectively.

In DNN processors, main memory consumes much more energy than arithmetic operations. Therefore, many memory-oriented network scheduling (MONS) techniques are introduced to exploit on-chip data reuse opportunities and reduce accesses to memory. However, to derive the theoretical lower bound of memory overhead for DNNs is still a significant challenge, which also sheds light on how to reach memory-level optimality by means of network scheduling. Prior work on MONS mainly focused on disparate optimization techniques or missed some of the data reusing opportunities in diverse network models, thus their results are likely to deviate from the true memory-optimality that can be achieved in processors. This paper introduces Olympus, which comprehensively considers the entire memory-level DNN scheduling space, formally analyzes the true memory-optimality and also how to reach the memory-optimal schedules for an arbitrary DNN running on a DNN processor. The key idea behind Olympus is to derive a true memory lower-bound regarding both the intra-layer and inter-layer reuse opportunities, which has not been simultaneously explored by prior works. Evaluation on SOTA DNN processors of different architectures shows that Olympus can guarantee the minimum off-chip memory access, and it reduces 12.3-85.6% DRAM access and saves 7.4-70.3% energy on the latest network models.

Bit-serial architectures (BSAs) are becoming increasingly popular in low power neural network processor (NNP) design. However, the performance and efficiency of state-of-the-art BSA NNPs are heavily depending on the distribution of ineffectual weight-bits of the running neural network. To boost the efficiency of third-party BSA accelerators, this work presents Bit-Pruner, a software approach to learn BSA-favored neural networks without resorting to hardware modifications. The techniques proposed in this work not only progressively prune but also structure the non-zero bits in weights, so that the number of zero-bits in the model can be increased and also load-balanced to suit the architecture of the target BSA accelerators. According to our experiments on a set of representative neural networks, Bit-Pruner increases the bit-sparsity up to 94.4% with negligible accuracy degradation. When the bit-pruned models are deployed onto typical BSA accelerators, the average performance is 2.1X and 1.5X higher than the baselines running non-pruned and weight-pruned networks, respectively.

Generative neural network is a new category of neural networks and it has been widely utilized in many applications such as content generation, unsupervised learning, segmentation, and pose estimation. It typically involves massive computing-intensive deconvolution operations that cannot be fitted to conventional neural network processors directly. However, prior works mainly investigated specialized hardware architectures through intensive hardware modifications to the existing deep learning processors to accelerate deconvolution together with the convolution. In contrast, this article proposes a novel deconvolution implementation with a software approach and enables fast and efficient deconvolution execution on the existing deep learning processors. Our proposed method reorganizes the computation of deconvolution and allows the deep learning processors to treat it as the standard convolution by splitting the original deconvolution filters into multiple small filters. Compared to prior acceleration schemes, the implemented acceleration scheme achieves 2.4× -4.3× performance speedup and reduces the energy consumption by 27.7 -54.5 percent on a set of realistic benchmarks. In addition, we have also applied the deconvolution computing approach to the off-the-shelf commodity deep learning processors. The performance of deconvolution also exhibits significant performance speedup over prior deconvolution implementations.

Unlike standard Convolutional Neural Networks (CNNs) with fully-connected layers, Fully Convolutional Neural Networks (FCN) are prevalent in computer vision applications such as object detection, semantic/image segmentation, and the most popular generative tasks based on Generative Adversarial Networks (GAN). In an FCN, traditional convolutional layers and deconvolutional layers contribute to the majority of the computation complexity. However, prior deep learning accelerator designs mostly focus on CNN optimization. They either use independent compute-resources to handle deconvolution or convert deconvolutional layers (Deconv) into general convolution operations, which arouses considerable overhead. To address this problem, we propose a unified fully convolutional accelerator aiming to handle both the deconvolutional and convolutional layers with a single processing element (PE) array. We re-optimize the conventional CNN accelerator architecture of regular 2D processing elements array, to enable it more efficiently support the data flow of deconvolutional layer inference. By exploiting the locality in deconvolutional filters, this architecture reduces the consumption of on-chip memory communication from 24.79 GB to 6.56 GB and improves the power efficiency significantly. Compared to prior baseline deconvolution acceleration scheme, the proposed accelerator achieves 1.3X–44.9X speedup and reduces the energy consumption by 14.60/0-97.6% on a set of representative benchmark applications. Meanwhile, it keeps similar CNN inference performance to that of an optimized CNN-only accelerator with negligible power consumption and chip area overhead.