Prior hardware accelerator designs primarily focused on single-chip solutions for 10 MB-class computer vision models. The GB-class transformer models for natural language processing (NLP) impose challenges on existing accelerator design due to the massive number of parameters and the diverse matrix multiplication (MatMul) workloads involved. This work proposes a heterogeneous 3D-based accelerator design for transformer models, which adopts an interposer substrate with multiple 3D memory/logic hybrid cubes optimized for accelerating different MatMul workloads. An approximate computing scheme is proposed to take advantage of heterogeneous computing paradigms of mixed-signal compute-in-memory (CIM) and digital tensor processing units (TPU). From the system-level evaluation results, 10 TOPS/W energy efficiency is achieved for the BERT and GPT2 model, which is about 2.6× ∼ 3.1× higher than the baseline with 7 nm TPU and stacked FeFET memory.

Emerging nonvolatile memories as embedded memories offer low leakage power and high memory density, compared to the static random access memory (SRAM) and embedded dynamic random access memory (eDRAM) at the same technology node. However, the emerging memories generally suffer from limited cycling endurance. For read/write intensive applications, the limited endurance could become a bottleneck that limits the lifetime of the overall system. In this work, Intel’s reported prototype 3-D stackable ferroelectric random access memory (FeRAM) is considered as the global buffer memory of a tensor-processing-unit (TPU)-like architecture. An endurance-aware compiler is proposed to evaluate the maximum number of deep neural network (DNN) trainings considering the experimentally measured endurance limit. In addition, the proposed compiler applies two strategies to alleviate the endurance issue. The first strategy is wear leveling, and the second strategy is the dual-mode operation between volatile and nonvolatile modes. The maximum numbers of trainings increase by$6\times $to$300\times $and$4\times $to$58\times $thanks to the wear-leveling and dual-mode operations, respectively. Finally, a guideline of the system endurance (maximum number of trainings) is provided with given memory device endurance to bridge the gap between memory device engineers and system designers.

After the success of the transformer networks on natural language processing (NLP), the application of transformers to computer vision has followed suit to deliver unprecedented performance gains on vision tasks including image recognition and object detection. The multi-head self-attention (MSA) is the key component in transformers, allowing the models to learn the amount of attention paid to each input position. In particular, hierarchical vision transformers (HVTs) utilize window-based MSA to capture the benefits of the attention mechanism at various scales for further accuracy enhancements. Despite its strong modeling capability, MSA involves complex operations that make transformers prohibitively costly for hardware deployment. Existing hardware accelerators have mainly focused on the MSA workloads in NLP applications, but HVTs involve different parameter dimensions, input sizes, and data reuse opportunities. Therefore, we design the RAWAtten architecture to target the window-based MSA workloads in HVT models. Each w-core in RAWAtten contains near-memory compute engines for linear layers, MAC arrays for intermediate matrix multiplications, and a lightweight reconfigurable softmax. The w-cores can be combined at runtime to perform hierarchical processing to accommodate varying model parameters. Compared to the baseline GPU, RAWAtten at 40nm provides 2.4x average speedup for running the window-MSA workloads in Swin transformer models while consuming only a fraction of GPU power. In addition, RAWAtten achieves 2x area efficiency compared to prior ASIC accelerator for window-MSA.

In this article, we propose a hardware accelerator design using ferroelectric transistor (FeFET)-based hybrid precision synapse (HPS) for deep neural network (DNN) on-chip training. The drain erase scheme for FeFET programming is incorporated for both FeFET HPS design and FeFET buffer design. By using drain erase, high-density FeFET buffers can be integrated onchip to store the intermediate input-output activations and gradients, which reduces the energy consuming off-chip DRAM access. Architectural evaluation results show that the energy efficiency could be improved by 1.2× ∼ 2.1×, 3.9× ∼ 6.0× compared to the other HPS-based designs and emerging non-volatile memory baselines, respectively. The chip area is reduced by 19% ∼ 36% compared with designs using SRAM on-chip buffer even though the capacity of FeFET buffer is increased. Besides, by utilizing drain erase scheme for FeFET programming, the chip area is reduced by 11% ∼ 28.5% compared with the designs using body erase scheme.

In this paper, a dual-mode buffer memory based on the CMOS compatible HfZrO2 ferroelectric material is proposed for DNN accelerators. It can operate in both volatile eDRAM mode and non-volatile ferroelectric RAM (FeRAM) mode. The functionality of the proposed dual-mode memory bit-cell design is verified using SPICE simulation with the multi-domain Preisach physical model. A data-lifetime-aware memory mode configuration protocol is proposed to optimize the buffer access energy for both DNN inference and training. Detailed circuitry and architectural support for the dual-mode memory are discussed. For DNN training with ferroelectric-field-effect-transistor (FeFET) and SRAM-based compute-in-memory (CIM) accelerator, the proposed dual-mode buffer design improves the overall energy efficiency by 92.2%~98.7%, 44.1%~47.6%, 12.6%~13.0% compared to baseline designs using SRAM buffer with the same buffer area, eDRAM and FeRAM with the same buffer capacity, respectively. For DNN inference with tensor-processing-unit (TPU)-like systolic array, the energy efficiency during computing is improved by 40.7%~45.6%, 18.4%~29.6% compared to the designs with eDRAM and FeRAM buffer, respectively. By storing the persistent data using the non-volatile mode, the energy efficiency of systolic array is improved by 2.3~5.5 over SRAM-based design when standby is frequent.

This article presents an algorithmic approach to design reliable deep neural networks (DNNs) in the presence of stochastic variations in the network parameters induced by process variations in the bit cells in a processing-in-memory (PIM) architecture. We propose and derive a Hessian-based sensitivity metric that can be computed without computing or storing the full Hessian to identify and protect the “important” network parameters while allowing large variations in unprotected parameters. We also show that this metric can be used to aggressively quantize unprotected network parameters in the PIM for improved inference efficiency and compute density. Experiments on modern DNNs like ResNet, MobileNetv2, and DenseNet on CIFAR10 using measured RRAM device data shows the effectiveness of our approach.

Compute-in-memory (CIM) is an attractive solution to address the “memory wall” challenges for the extensive computation in machine learning hardware accelerators. Prior CIM-based architectures, though can adapt to different neural network models during the design time, they are implemented to different custom chips. Therefore, a specific chip instance is restricted to a specific network during runtime. However, the development cycle of the hardware is normally far behind the emergence of new algorithms. In this paper, a runtime reconfigurable design methodology of CIM-based accelerator is proposed to support a class of convolutional neural networks running on one pre-fabricated chip instance. First, several design aspects are investigated: 1) reconfigurable weight mapping method; 2) input side of data transmission, mainly about the weight reloading; 3) output side of data processing, mainly about the reconfigurable accumulation. Then, system-level performance benchmark is performed for the inference of different models like VGG-8 on CIFAR-10 dataset and AlexNet, GoogLeNet, ResNet-18 and DenseNet-121 on ImageNet dataset to measure the tradeoffs between runtime reconfigurability, chip area, memory utilization, throughput and energy efficiency.

Deep neural network (DNN) training produces a large amount of intermediate data. As off-chip DRAM access is both energy and time consuming, sufficient on-chip buffer is preferred to achieve high energy efficiency for DNN accelerator designs. However, the low integration density and high leakage current of SRAM lead to large area cost and high standby power. The frequent refresh of embedded DRAM (eDRAM) degrades the energy efficiency due to its short refresh interval (40~100µs). In this paper, a dual-mode buffer memory that can operate in both volatile eDRAM mode and non-volatile ferroelectric RAM (FeRAM) mode is proposed, which is based on the CMOS compatible HfZr02 material. The functionality of the proposed dual-mode memory design is verified using SPICE simulation with the multi-domain Preisach model. A data lifetime-aware memory mode configuration protocol is proposed to optimize the buffer access energy. The architectural benchmark for DNN training shows 33.8%, 17.1 % and 109.4% higher energy efficiency than baseline designs with eDRAM, FeRAM and SRAM with the same buffer area, respectively. The chip standby power is reduced by 26.8x~47.5x and 1.5x~10.6x compared with the SRAM and eDRAM baselines. The chip area overhead of the dual-mode buffer design is 5.7%.

Machine learning inference engine is of great interest to smart edge computing. Compute-in-memory (CIM) architecture has shown significant improvements in throughput and energy efficiency for hardware acceleration. Emerging nonvolatile memory (eNVM) technologies offer great potentials for instant on and off by dynamic power gating. Inference engine is typically pre-trained by the cloud and then being deployed to the field. There is a new security concern on cloning of the weights stored on eNVM-based CIM chip. In this paper, we propose a countermeasure to the weight cloning attack by exploiting the process variations of the periphery circuitry. In particular, we use weight fine-tuning to compensate the analog-to-digital converter (ADC) offset for a specific chip instance while inducing significant accuracy drop for cloned chip instances. We evaluate our proposed scheme on a CIFAR-10 classification task using a VGG- 8 network. Our results show that with precisely chosen transistor size on the employed SAR-ADC, we could maintain 88%~90% accuracy for the fine-tuned chip while the same set of weights cloned on other chips will only have 20~40% accuracy on average. The weight fine-tune could be completed within one epoch of 250 iterations. On average only 0.02%, 0.025%, 0.142% of cells are updated for 2-bit, 4-bit, 8-bit weight precisions in each iteration.

As AI applications become pervasive on edge device, incrementally learning new tasks is demanded for deep neural network (DNN) models. In this article, we proposed AILC, a compute-in-memory (CIM)-based accelerator for on-chip incremental learning using STT-MRAM technology. On the software side, a network-expansion-based low-precision training algorithm is proposed for incremental learning, where the loss function is modified to handle the unbalanced training dataset. On the hardware side, the detailed CIM accelerator design for incremental learning is illustrated. A workload-aware hardware resources assignment protocol is proposed to improve the throughput when the workload of weight gradient calculation is low. The software simulation results on CIFAR-100 dataset show that the proposed algorithm can effectively support incremental learning despite the device conductance variation exists. System-level benchmark shows that AILC could achieve 147×, 3.7×~28.7×, 2.05×~2.9× higher energy efficiency than Nvidia Titan-V GPU, RRAM-based CIM accelerators and edge TPU/GPU, respectively. Compared to the baselines, the throughput of AILC is improved by 2.0×~2.2× on average with the hardware resources assignment protocol, which results in 4.1×~21.4× higher throughput than edge TPU/GPU.

To efficiently deploy machine learning applications to the edge, compute-in-memory (CIM) based hardware accelerator is a promising solution with improved throughput and energy efficiency. Instant-on inference is further enabled by emerging non-volatile memory technologies such as resistive random access memory (RRAM). This paper reviews the recent progresses of the RRAM based CIM accelerator design. First, the multilevel states RRAM characteristics are measured from a test vehicle to examine the key device properties for inference. Second, a benchmark is performed to study the scalability of the RRAM CIM inference engine and the feasibility towards monolithic 3D integration that stacks RRAM arrays on top of advanced logic process node. Third, grand challenges associated with in-situ training are presented. To support accurate and fast in-situ training and enable subsequent inference in an integrated platform, a hybrid precision synapse that combines RRAM with volatile memory (e.g. capacitor) is designed and evaluated at system-level. Prospects and future research needs are discussed.

Compute-in-memory (CIM) is an attractive solution to address the “memory wall” challenges for the extensive computation in deep learning hardware accelerators. For custom ASIC design, a specific chip instance is restricted to a specific network during runtime. However, the development cycle of the hardware is normally far behind the emergence of new algorithms. Although some of the reported CIM-based architectures can adapt to different deep neural network (DNN) models, few details about the dataflow or control were disclosed to enable such an assumption. Instruction set architecture (ISA) could support high flexibility, but its complexity would be an obstacle to efficiency. In this article, a runtime reconfigurable design methodology of CIM-based accelerators is proposed to support a class of convolutional neural networks running on one prefabricated chip instance with ASIC-like efficiency. First, several design aspects are investigated: (1) the reconfigurable weight mapping method; (2) the input side of data transmission, mainly about the weight reloading; and (3) the output side of data processing, mainly about the reconfigurable accumulation. Then, a system-level performance benchmark is performed for the inference of different DNN models, such as VGG-8 on a CIFAR-10 dataset and AlexNet GoogLeNet, ResNet-18, and DenseNet-121 on an ImageNet dataset to measure the trade-offs between runtime reconfigurability, chip area, memory utilization, throughput, and energy efficiency.

Compute-in-memory (CIM) with emerging non-volatile memories (eNVMs) is time and energy efficient for deep neural network (DNN) inference. However, challenges still remain for in-situ DNN training with eNVMs due to the asymmetric weight update behavior, high programming latency and energy consumption. To overcome these challenges, a hybrid precision synapse combining eNVMs with capacitor has been proposed. It leverages the symmetric and fast weight update in the volatile capacitor, as well as the non-volatility and large dynamic range of the eNVMs. In this paper, in-situ DNN training architecture with hybrid precision synapses is proposed and benchmarked with the modified NeuroSim simulator. First, all the circuit modules required for in-situ training with hybrid precision synapses are designed. Then, the impact of weight transfer interval and limited capacitor retention time on training accuracy is investigated by incorporating hardware properties into Tensorflow simulation. Finally, a system-level benchmark is conducted for hybrid precision synapse compared with baseline design that is solely based on eNVMs.

STT-MRAM is a promising candidate as embedded non-volatile memory (NVM) at 28nm and beyond. Due to its limited on/off ratio, STT-MRAM is often used as digital memory that only allows row-by-row read-out for near-memory computing. This work proposes design strategies to overcome this limitation with a new bit-cell design to enable parallel read-out for in-memory computing, which is of great interests for deep neural network (DNN) acceleration. We consider the non-ideal device properties that degrade inference accuracy including small on/off ratio, cell-to-cell MTJ conductance variation and current sense amplifier (CSA) offset. We propose three techniques to minimize inference accuracy degradation: 1) a 2T-2MTJ bit-cell design with high on/off ratio, 2) redundancy for MSB weights to mitigate the impact of MTJ conductance variations, and 3) a hybrid-layer mapping scheme to reduce column current thus mitigating CSA offset effect. DNN benchmarking results show that on CIFAR-10 dataset, the inference accuracy can be maintained at > 90% in the presence of 10% MTJ conductance variations, and >87.5% after considering CSA offset effect, with minimal 8% energy and 4% chip area overhead.

Compute-in-memory (CIM) with emerging non-volatile memories (eNVMs) is time and energy efficient for deep neural network (DNN) inference. However, challenges still remain for DNN in-situ training with eNVMs due to the asymmetric weight update behavior, high programming latency and energy consumption. To overcome these challenges, a hybrid precision synapse combining eNVMs with capacitor has been proposed. It leverages the symmetric and fast weight update in the volatile capacitor, as well as the non-volatility and large dynamic range of the eNVMs. In this article, DNN in-situ training architecture with hybrid precision synapses is proposed and system level benchmarked is conducted. First, the circuit modules required for in-situ training with hybrid precision synapses are designed and the system architecture is proposed. Then, the impact of different weight precision configurations, weight transfer interval and limited capacitor retention time on training accuracy is investigated by incorporating hardware properties into Tensorflow simulation. Finally, the system-level benchmark is conducted at 32nm technology node in the modified NeuroSim simulator for hybrid precision synapse, in comparison with the baseline designs that are solely based on eNVMs or SRAM technology. The benchmark results show that CIM accelerator based on hybrid precision synapse achieves at least 3.07x and 2.89x better energy efficiency for training compared with its eNVM counterparts and SRAM technology at 32nm node, respectively. 227x and 33.8x better energy efficiency are obtained when compared to GPU and TPU. The scaling trend of hybrid precision synapse is projected towards 7nm node and comparison with state-of-the-art 7nm SRAM technology is made.

Compute-in-memory (CIM) is a promising computing paradigm to accelerate the inference of deep neural network (DNN) algorithms due to its high processing parallelism and energy efficiency. Prior CIM-based DNN accelerators mostly consider full custom design, which assumes that all the weights are stored on-chip. For lightweight smart edge devices, this assumption may not hold. In this article, CIM-based DNN accelerators are designed and benchmarked under different chip area constraints. First, a scheduling strategy and dataflow for DNN inference is investigated when only part of the weights can be stored on-chip. Two weight reload schemes are evaluated: 1) reload partial weights and reuse input/output feature maps and 2) load a batch of input and reuse the partial weights on-chip across the batch. Then, system-level performance benchmark is performed for the inference of ResNet-18 on ImageNet data set. The design tradeoffs with different area constraints, dataflow, and device technologies [static random access memory (SRAM) versus ferroelectric field-effect transistor (FeFET)] are discussed.