Non-linear operations such as GELU, Layer normalization, and Soft-max are essential yet costly building blocks of Transformer models. Several prior works simplified these operations with look-up tables or integer computations, but such approximations suffer inferior accuracy or considerable hardware cost with long latency. This paper proposes an accurate and hardware-friendly approximation framework for efficient Transformer inference. Our framework employs a simple neural network as a universal approximator with its structure equivalently transformed into a Look-up table(LUT). The proposed framework called Neural network generated LUT(NN-LUT) can accurately replace all the non-linear operations in popular BERT models with significant reductions in area, power consumption, and latency.

The long short-term memory (LSTM) is a widely used neural network model for dealing with time-varying data. To reduce the memory requirement, pruning is often applied to the weight matrix of the LSTM, which makes the matrix sparse. In this paper, we present a new sparse matrix format, named rearranged compressed sparse column (RCSC), to maximize the inference speed of the LSTM hardware accelerator. The RCSC format speeds up the inference by: 1) evenly distributing the computation loads to processing elements (PEs) and 2) reducing the input vector load miss within the local buffer. We also propose a hardware architecture adopting hierarchical input buffer to further reduce the pipeline stalls which cannot be handled by the RCSC format alone. The simulation results for various datasets show that combined use of the RSCS format and the proposed hardware requires 2× smaller inference runtime on average compared to the previous work.

The probabilistic Bayesian inference of real-time input data is becoming more popular, and the importance of semisupervised learning is growing. We present a classification restricted Boltzmann machine (ClassRBM)-based hardware accelerator with on-chip semisupervised learning and Bayesian inference capability. ClassRBM is a specific type of Markov network that can perform classification tasks and reconstruct its input data. ClassRBM has several advantages in terms of hardware implementation compared to other backpropagation-based neural networks. However, its accuracy is relatively low compared to backpropagation-based learning. To improve the accuracy of ClassRBM, we propose the multi-neuron-per-class (multi-NPC) voting scheme. We also reveal that the contrastive divergence (CD) algorithm, which is commonly used to train RBM, shows poor performance in this multi-NPC ClassRBM. As an alternative, we propose an asymmetric contrastive divergence (ACD) training algorithm that improves the accuracy of multi-NPC ClassRBM. With the ACD learning algorithm, ClassRBM operates in the form of a combination of Markov Chain training and Bayesian inference. The experimental results on a field-programmable gate array (FPGA) board for a Modified National Institute of Standards and Technology data set confirm that the inference accuracy of the proposed ACD algorithm is 5.82% higher for a supervised learning case and 12.78% higher for a 1% labeled semisupervised learning case than the conventional CD algorithm. Also, the GeCo ver.2 hardware implemented on a Xilinx ZCU102 FPGA board was 349.04 times faster than the C simulation on CPU.

In this paper, we present an integrated solution to design a high-performance LSTM accelerator. We propose a fast and flexible hardware architecture, named Peregrine, supported by a stack of innovations from algorithm to hardware design. Peregrine first minimizes the memory footprint by limiting the synaptic connection patterns within the LSTM network. Also, Peregrine provides parallel Huffman decoders with adaptive clocking to provide flexibility in dealing with a wide range of sparsity levels in the weight matrices. All these features are incorporated in a novel hardware architecture to maximize energy-efficiency. As a result, Peregrine improves performance by ~38% and energy-efficiency by ~33% in speech recognition compared to the state-of-the-art LSTM accelerator.

The LSTM is a popular neural network model for modeling or analyzing the time-varying data. The main operation of LSTM is a matrix-vector multiplication and it becomes sparse (spMxV) due to the widely-accepted weight pruning in deep learning. This paper presents a new sparse matrix format, named CBSR, to maximize the inference speed of the LSTM accelerator. In the CBSR format, speed-up is achieved by balancing out the computation loads over PEs. Along with the new format, we present a simple network transformation to completely remove the hardware overhead incurred when using the CBSR format. Also, the detailed analysis on the impact of network size or the number of PEs is performed, which lacks in the prior work. The simulation results show 16~38% improvement in the system performance compared to the well-known CSC/CSR format. The power analysis is also performed in 65nm CMOS technology to show 9~22% energy savings.

We present a Classification Restricted Boltzmann Machine (Class-RBM) hardware for embedded machines with on-chip learning capability. The RBM is a kind of the generative model, and has been used as one of the most popular feature extractors and image pre-processors. The ClassRBM is a variant of the RBM that is adapted to classification tasks. We propose the multi-Neuron-Per-Class (multi-NPC) voting scheme for improving accuracy of ClassRBM. We also show that the Contrastive Divergence (CD), which is one of the most popular algorithms to train RBM, has limitations in multi-NPC ClassRBM learning and propose a modified CD algorithm to overcome the limitation. Experimental results on FPGA flatform for MNIST datasets confirm that classification accuracy of the proposed algorithm is ∼ 2.12% higher than the conventional CD.