Analog non-volatile memory (NVM)-based accelerators for Deep Neural Networks (DNNs) can achieve high-throughput and energy-efficient multiply-accumulate (MAC) operations by taking advantage of massively parallelized analog compute, implemented with Ohm's law and Kirchhoff's current law on arrays of resistive memory devices. Competitive end-to-end DNN accuracies can be obtained, provided that weights are accurately programmed onto NVM devices and MAC operations are sufficiently linear. In this paper, we report architectural and circuit advances for such Analog NVM-based accelerators. We describe a highly heterogeneous and programmable accelerator architecture for DNN inference that combines analog NVM memory-array “Tiles” for weight-stationary, energy-efficient MAC operations, together with heterogeneous special-function Compute-Cores for auxiliary digital computation. Massively parallel vectors of neuron-activation data are exchanged over short distances using a dense and efficient circuit-switched 2D mesh, enabling a wide range of DNN workloads, including CNNs, LSTMs, and Transformers. We also show a 14-nm inference chip consisting of multiple$\mathbf{512}\times \mathbf{512}$arrays of Phase Change Memory (PCM) devices which implements multiple DNN benchmarks using such a circuit-switched 2D mesh.

We introduce a highly heterogeneous and programmable compute-in-memory (CIM) accelerator architecture for deep neural network (DNN) inference. This architecture combines spatially distributed CIM memory array “tiles” for weight-stationary, energy-efficient multiply–accumulate (MAC) operations, together with heterogeneous special-function compute cores for auxiliary digital computation. Massively parallel vectors of neuron activation data are exchanged over short distances using a dense and efficient circuit-switched 2-D mesh, offering full end-to-end support for a wide range of DNN workloads, including CNNs, long-short-term-memory (LSTM), and transformers. We discuss the design of the “analog fabric”—the 2-D grid of tiles and compute cores interconnected by the 2-D mesh—and address the efficiency in both mapping of DNNs onto the hardware and in pipelining of various DNN workloads across a range of batch sizes. We show, for the first time, system-level assessments using projected component parameters for a realistic “analog AI” system, based on dense crossbar arrays of low-power nonvolatile analog memory elements, while incorporating a single common analog fabric design that can scale to large networks by introducing data transport between multiple analog AI chips. Our performance estimates for several networks, including large LSTM and bidirectional encoder representations from transformers (BERT), show highly competitive throughput while offering$40\times $–$140\times $higher energy efficiency than NVIDIA A100—thus illustrating the strong promise of analog AI and the proposed architecture for DNN inference applications.

Analog non-volatile memory (NVM)-based accelerators for deep neural networks perform high-throughput and energy-efficient multiply-accumulate (MAC) operations (e.g., high TeraOPS/W) by taking advantage of massively parallelized analog MAC operations, implemented with Ohm’s law and Kirchhoff’s current law on array-matrices of resistive devices. While the wide-integer and floating-point operations offered by conventional digital CMOS computing are much more suitable than analog computing for conventional applications that require high accuracy and true reproducibility, deep neural networks can still provide competitive end-to-end results even with modest (e.g., 4-bit) precision in synaptic operations. In this paper, we describe a 14-nm inference chip, comprising multiple 512$\times$ 512 arrays of Phase Change Memory (PCM) devices, which can deliver software-equivalent inference accuracy for MNIST handwritten-digit recognition and recurrent LSTM benchmarks, by using compensation techniques to finesse analog-memory challenges such as conductance drift and noise. We also project accuracy for Natural Language Processing (NLP) tasks performed with a state-of-art large Transformer-based model, BERT, when mapped onto an extended version of this same fundamental chip architecture.

By performing parallelized multiply-accumulate operations in the analog domain at the location of weight data, crossbar-array "tiles" of analog non-volatile memory (NVM) devices can potentially accelerate the forward-inference of deep neural networks. To be successful, such systems will need to achieve two related but challenging goals. First is the achievement of high neural network classification accuracies, indistinguishable from those achieved with conventional approaches, despite the difficulties of programming NVM devices accurately in the presence of significant device-to-device variability. Towards this first goal, we describe row-wise Phase-Change Memory (PCM) programming schemes for rapid yet accurate weight- programming. The second goal is highly energy-efficient forward- inference of multi-layer neural networks, requiring efficiency in both the massively-parallel analog-AI operations performed at each tile, as well as efficiency in how the resulting neuron-excitation data vectors get conveyed from tile to tile. Towards this second goal, micro-architectural design ideas including source-follower-based readout, array segmentation, and transmit-by- duration are described.

Neuromorphic computation based on analog nonvolatile memories (NVMs) holds great promise to improve Deep Neural Networks inference performance. In virtue of an architecture that executes the computation at the location of the stored weight data, remarkable gains in energy efficiency and speed are projected over competing von Neumann architectures leveraged by existing digital accelerators. Here we describe two optimization strategies for NVMs: one for programming the memory elements and one of cell design, both aimed at mitigating the effect of NVM non-idealities on the performance of analog, phase-change memory-based accelerators. We then demonstrate the advantages realized by such strategies on the inference accuracy of Long Short Term Memory networks and evaluate the energy requirements of such networks.

Cognitive computing describes “systems that learn at scale, reason with purpose, and interact with humans naturally” [1]. In this paper, we review our work towards enabling “next generation” cognitive computing using neuromorphic computational schemes that could potentially outperform present-day CPUs and GPUs. Here we use large arrays of Resistive Non-Volatile Memories (NVM) with device conductance serving as synaptic weight. We focus on training and classification using fully-connected networks based on the backpropagation algorithm, and show that our approach could offer power and speed advantages over conventional Von-Neumann processors. We also propose some circuit approximations that improve network parallelism without significantly degrading classification accuracy. Finally, we explore the requirements for a system implementation of on-chip learning.

Machine Learning (ML) is an attractive application of Non-Volatile Memory (NVM) arrays [1,2]. However, achieving speedup over GPUs will require minimal neuron circuit sharing and thus highly area-efficient peripheral circuitry, so that ML reads and writes are massively parallel and time-multiplexing is minimized [2]. This means that neuron hardware offering full `software-equivalent' functionality is impractical. We analyze neuron circuit needs for implementing back-propagation in NVM arrays and introduce approximations to reduce design complexity and area. We discuss the interplay between circuits and NVM devices, such as the need for an occasional RESET step, the number of programming pulses to use, and the stochastic nature of NVM conductance change. In all cases we show that by leveraging the resilience of the algorithm to error, we can use practical circuit approaches yet maintain competitive test accuracies on ML benchmarks.

Training of large-scale deep neural networks is often constrained by the available computational resources. We study the effect of limited precision data representation and computation on neural network training. Within the context of low-precision fixed-point computations, we observe the rounding scheme to play a crucial role in determining the network’s behavior during training. Our results show that deep networks can be trained using only 16-bit wide fixed-point number representation when using stochastic rounding, and incur little to no degradation in the classification accuracy. We also demonstrate an energy-efficient hardware accelerator that implements low-precision fixed-point arithmetic with stochastic rounding

Emerging nanodevice-based architectures will be impacted by parameter variation in conjunction with high defect rates. Variations in key physical parameters are caused by manufacturing imprecision as well as fundamental atomic scale randomness. In this article, the impact of parameter variation on nanoscale computing fabrics is extensively studied through a novel integrated methodology across device, circuit and architectural levels. This integrated approach enables to study in detail the impact of physical parameter variation across all fabric layers. A final contribution of the article includes novel techniques to address this impact. The variability framework, while generic, is explored extensively on the Nanoscale Application Specific Integrated Circuits (NASICs) nanowire fabric. For variation of σ = 10 in key physical parameters, the on current is found to vary by up to 3.5X. Circuit-level delay shows up to 118% deviation from nominal. Monte Carlo simulations using an architectural simulator found 67% nanoprocessor chips to operate below nominal frequencies due to variation. New built-in variation mitigation and fault-tolerance schemes, leveraging redundancy, asymmetric delay paths and biased voting schemes, were developed and evaluated to mitigate these effects. They are shown to improve performance by up to 7.5X on a nanoscale processor design with variation, and improve performance in designs relying on redundancy for defect tolerance, without variation assumed. Techniques show up to 3.8X improvement in effective-yield performance products even at a high 12% defect rate. The suite of techniques provides a design space across key system-level metrics such as performance, yield and area.

Extensive research has been done in prefetching techniques that hide memory latency in microprocessors leading to performance improvements. However, the energy aspect of prefetching is relatively unknown. While aggressive prefetching techniques often help to improve performance, they increase energy consumption by as much as 30% in the memory system. This paper provides a detailed evaluation on the energy impact of hardware data prefetching and then presents a set of new energy-aware techniques to overcome prefetching energy overhead of such schemes. These include compiler-assisted and hardware-based energy-aware techniques and a new power-aware prefetch engine that can reduce hardware prefetching related energy consumption by 7-11 ×. Combined with the effect of leakage energy reduction due to performance improvement, the total energy consumption for the memory system after the application of these techniques can be up to 12% less than the baseline with no prefetching.