In-situ training architectures based on resistive random-access memory (ReRAM) have attracted significant attention due to their exceptional energy efficiency. However, the inherent stochasticity of ReRAM devices severely degrades training convergence. To address this challenge, this work proposes a ReRAM-based in-situ training accelerator (ReBIT) architecture. The ReBIT integrates ReRAM with static random-access memory (SRAM) devices, leveraging the deterministic characteristics of SRAM-based computations to suppress the inherent stochasticity of ReRAM devices. Experimental results demonstrate that the ReBIT architecture achieves convergence performance comparable to full-precision software training.

Large language models (LLMs) have demonstrated impressive capabilities across a wide range of applications, but demand substantial memory and compute resources during inference. Existing quantization methods expose a trade-off between efficiency and accuracy: weight-only quantization (WOQ) incurs costly dequantization overheads, while integer weight-and-activation quantization (INT-WAQ) reduces precision and degrades model quality. Non-uniform weight-and-activation quantization (NU-WAQ) can better capture the non-uniform distributions of LLM weights and activations, yet remains incompatible with conventional low-precision compute units. This paper presents OASIS, a lookup table (LUT)-based architecture that enables efficient general matrix multiplication (GEMM) between non-uniformly quantized weights and activations without requiring dequantization. OASIS employs pre-computed Cartesian Product LUTs, achieving a 64x reduction in LUT size and enabling a 1024x higher computational parallelism over existing LUT-based GEMM methods. To preserve accuracy under aggressive activation quantization, OASIS introduces an outlier-aware quantization scheme with concurrent LUT-based GEMM and error compensation for outliers. Furthermore, we design Orizuru, an efficient top-k detection engine for real-time activation outlier identification. According to extensive evaluations, OASIS incurs an average accuracy drop of only 1.98% compared to the FP16 baseline, which is 5.18% lower than Atom. On the hardware side, OASIS achieves an average 3.00x speedup and a 1.44x energy efficiency improvement compared to the FIGLUT accelerator.

Large language models (LLMs) are widely used for natural language understanding and text generation. An LLM model relies on a time-consuming step called LLM decoding to generate output tokens. Several prior works focus on improving the performance of LLM decoding using parallelism techniques, such as batching and speculative decoding. State-of-the-art LLM decoding has both compute-bound and memory-bound kernels. Some prior works statically identify and map these different kernels to a heterogeneous architecture consisting of both processing-in-memory (PIM) units and computation-centric accelerators (e.g., GPUs). We observe that characteristics of LLM decoding kernels (e.g., whether or not a kernel is memory-bound) can change dynamically due to parameter changes to meet user and/or system demands, making (1) static kernel mapping to PIM units and computation-centric accelerators suboptimal, and (2) one-size-fits-all approach of designing PIM units inefficient due to a large degree of heterogeneity even in memory-bound kernels.

Computing-in-memory (CIM) architectures demonstrate superior performance over traditional architectures. To unleash the potential of CIM accelerators, many compilation methods have been proposed, focusing on application scheduling optimization specific to CIM. However, existing compilation methods often overlook CIM's capability to switch dynamically between compute and memory modes, which is crucial for accommodating the diverse memory and computational needs of real-world deep neural network architectures, especially the emerging large language models. To fill this gap, we introduce CMSwitch, a novel compiler to optimize resource allocation for CIM accelerators with adaptive mode-switching capabilities, thereby enhancing the performance of DNN applications. Specifically, our approach integrates the compute-memory mode switch into the CIM compilation optimization space by introducing a new hardware abstraction attribute. Then, we propose a novel compilation optimization pass that identifies the optimal network segment and the corresponding mode resource allocations using dynamic programming and mixed-integer programming. CMSwitch uses the tailored meta-operator to express the compilation result in a generalized manner. Evaluation results demonstrate that CMSwitch achieves an average speedup of 1.31x compared to existing SOTA CIM compilation works, highlighting CMSwitch's effectiveness in fully exploiting the potential of CIM processors for a wide range of real-world DNN applications.

In recent years, various computing-in-memory (CIM) processors have been presented, showing superior performance over traditional architectures. To unleash the potential of various CIM architectures, such as device precision, crossbar size, and crossbar number, it is necessary to develop compilation tools that are fully aware of the CIM architectural details and implementation diversity. However, due to the lack of architectural support in current popular open-source compiling stacks such as TVM, existing CIM designs either manually deploy networks or build their own compilers, which is time-consuming and labor-intensive. Although some works expose the specific CIM device programming interfaces to compilers, they are often bound to a fixed CIM architecture, lacking the flexibility to support the CIM architectures with different computing granularity. On the other hand, existing compilation works usually consider the scheduling of limited operation types (such as crossbar-bound matrix-vector multiplication). Unlike conventional processors, CIM accelerators are featured by their diverse architecture, circuit, and device, which cannot be simply abstracted by a single level if we seek to fully explore the advantages brought by CIM.

Quantization is one of the most hardware-efficient ways to reduce inference costs for deep neural network (DNN) models. Nevertheless, with the continuous increase of DNN model sizes (240× in two years) and the emergence of large language models, existing static quantization methods fail to utilize the sparsity and redundancy of models sufficiently. Motivated by the pervasive dynamism in data tensors across DNN models, we propose a dynamic precision quantization algorithm to further reduce computational costs beyond statically quantized DNN models. Furthermore, we find that existing precision-flexible accelerators cannot support the DNN models with dynamic precision. To this end, we design a novel accelerator, Drift, and achieve online scheduling to efficiently support dynamic precision execution. We conduct experiments with various DNN models, including CNN-based and Transformer-based models. Evaluation results show that Drift achieves 2.85× speedup and 3.12× energy saving compared to existing precision-flexible accelerators with statically quantized models.

ReRAM-based accelerator of bit-slicing architecture is a promising solution to neural network inference, which allows ineffectual bit-operation removal for greater potential gains. However, existing techniques mostly exploit the removal of weight-associated ineffectual operations, which cannot eliminate the activation-induced ineffectual operations. Alternatively, some approaches adopt an isolated two-stage approach to remove at the weight and activation-level, which leaves a big proportion of ineffectual bit-level operations. Therefore, in contrast to all these coarse-grained operation removal techniques, it is challenging to jointly eliminate ineffectual bit-operation induced by either activation or weight bit-slices for ReRAM-based accelerators. This work presents a novel ineffectual bit-operation removal approach and the accompanied ReRAM-based bit-operation clipping architecture that skips all those bit-level operations that make negligible impacts on neural network outputs. In experiments, the proposed bit-operation clipping ReRAM accelerator, Bit-Trimmer, achieves$5.28 \times$energy efficiency and$2.04 \times$speedup on average. Besides, compared with two SOTA ReRAM accelerator designs with bit-operation removal, it outperforms by$1.56 \times$and$1.88 \times$energy efficiency.

ReRAM-based Computing-in-Memory (CiM) architecture has been considered a promising solution to high-efficiency neural network accelerator, by conducting in-situ matrix multiplications and eliminating the movement of neural parameters from off-chip memory to computing units. However, we observed specific features of Graph Convolutional Network (GCN) tasks pose design challenges to implement a high-efficiency ReRAM GCN accelerator. The ultra-large input feature data in some GCN tasks incur massive data movements, the extremely sparse adjacency matrix and input feature data involve the valid computation, and the super-large adjacency matrix that exceeds available ReRAM capacity causes frequent expensive write operations. To address the above challenges, we propose TARe, a Task-Adaptive CiM architecture, which consists of a hybrid in-situ computing mode to support the input feature in crossbar computing, a compact mapping scheme for efficient sparse matrix computing, and a write-free mapping to eliminate write activities in the computations with the super-large adjacency matrix. Additionally, TARe is facilitated with a task adaptive selection algorithm to generate optimized design schemes for graph neural network tasks that have various operand sizes and data sparsity. We evaluate TARe on 11 diverse graph neural network tasks and compare it with different design counterparts, and the results show that achieves 168.06× speedup and 10.95× energy consumption reduction on average over the baseline in common graph convolutional network workloads.

Solving systems of linear equations is a fundamental problem in scientific computing, which has been extensively researched for decades. One of the most well-known solvers is Algebraic Multigrid (AMG), which is widely used in high performance computing due to its good scalability. But currently accelerating AMG relies on the traditional von Neumann architecture of storage and computation separation, which leads to a large data transmission overhead. In this work, we propose a ReRAM-based processing-in-memory (PIM) architecture named AmgR, which overcomes the limitations of the traditional von Neumann architecture for AMG acceleration.However, accelerating AMG on ReRAM is non-trivial, because (1) AMG has many computing kernels of various types; (2) there are irregular operations that cannot be directly performed using matrix-vector multiplication suitable for ReRAM, i.e., aggregation operation; (3) ReRAM has poor write endurance, and a lot of data during AMG acceleration needs to be rewritten into ReRAM, resulting in high write cost. To address these issues, firstly, we propose a flexible architecture, which can realize each kernel of AMG and is reused by many kernels to improve resource utilization. Secondly, we propose a dedicated unit to realize the aggregation operation. Finally, we present a new mapping strategy to greatly reduce the number of data handling and writes. The experimental results show that the performance of AmgR is improved by an average of one and two orders of magnitude compared to HYPRE on the CPU and AmgX on the GPU, respectively, while the energy consumption is reduced by an average of two and three orders of magnitude.

The emerging Resistive RAM (ReRAM) technology significantly boosts the performance and the energy efficiency of the deep learning accelerators (DLAs) via the Computing-in-Memory (CiM) architecture. However, ReRAM-based DLA also suffers a high occurrence rate of memory faults. How to detect and protect against the faults in ReRAM devices poses great challenges to ReRAM-based DLA design. In this work, we propose RRAMedy, an in-situ fault detection and network remedy framework for ReRAM-based DLAs. With the proposed Adversarial Example Testing, which is a lifetime on-device and on-line fault detection technique, it achieves high detection coverage of both hard faults and soft faults at a low run-time cost. In addition, it employs an edge-cloud collaborative model retraining method to tolerate the detected faults by leveraging the inherent fault-adaptive capability of DNNs. Meanwhile, to enable in-situ model remedy when the cloud assistance is absent due to security or overhead issues, we propose to accelerate the fault-masking retraining process on edge devices with parallelized Knowledge Transfer. Our experimental results show that the proposed fault detection technique achieves high fault detection accuracy and delivers real-time testing performance. Meanwhile, the proposed retraining approach greatly alleviates the accuracy degradation problem and achieves excellent performance speedups over the baselines.

ReRAM-based accelerators have shown great potential in neural network acceleration via in-memory analog computing. However, high-precision analog-to-digital converters (ADCs), which are required by the ReRAM crossbars to achieve high-accuracy network model inference, play an essential role in the energy-efficiency of the accelerators. Based on the discovery that the ADC precision requirements of crossbars are different, we propose the model-aware crossbarwise ADC precision assignment and the accompanied information-lossless low-bit ADCs to reduce energy overhead without sacrificing model accuracy. In experiments, the proposed information-lossless ReRAM accelerator, InfoX, only consumes 8.97% ADC energy of the SOTA baseline with no accuracy degradation at all.

In-memory computing (IMC) is recognized as one of the most promising architecture solution to realize energy-efficient neural network inference. Amongst many memory technology, resistive RAM (RRAM) is a very attractive device to implement the IMC-based neural network accelerator architecture, which is particularly suitable for power-constrained IoT systems. Due to the nature of low leakage and in-situ computing, the dynamic power consumption of dot-production operations in RRAM crossbars dominates the chip power, especially when applied to low-precision neural networks. This work investigates the correlation between the cell resistance state and the crossbar operation power, and proposes a state-aware RRAM accelerator (SARA) architecture for energy-efficient low-precision neural networks. With the proposed state-aware network training and mapping strategy, crossbars in the RRAM accelerator can perform in a lower power state. Furthermore, we also leverage the proposed RRAM accelerator architecture to reduce the power consumption of high-precision network inference with both single-level or multilevel RRAM. The evaluation results show that for binary neural networks, our design saves 40.53% RRAM computing energy on average over the baseline. For high precision neural networks, the proposed method reduces 11.67% computing energy on average without any accuracy loss.

Due to the nonvolatility nature of resistive RAM (ReRAM), dynamic operations in the arrays contribute to a much larger portion of power in ReRAM-based neural networks than static power. To reduce the dynamic power ofin-situoperations with neural parameters, precision-tuning is considered a viable approach of approximate computing to tradeoff excessive computation exactness for power and efficiency gains. However, the switching overhead of precision tuning in hardware severely impacts its effectiveness when the systems need to quickly react to the change of environment, user constraint or input quality. This work for the first time investigates the feasibility of agile precision tuning for neural network accelerators to benefit from approximate computing. The proposed computing in memory (CiM) CNN accelerators fully utilize the normally off characteristics of memristor crossbars to achieve instant network precision tuning without worrying about the model reloading penalty. The ReRAM-based accelerator, with the proposed neural parameter mapping policy and the novel mixed-model training method, induces negligible precision-switching latency and power consumption when compared with traditional variable precision accelerators. In evaluation with state-of-the-art workloads, the proposed ReRAM deep learning and neural network architecture saves 58.3%–62.47% area overhead over the baseline design. We also leverage the proposed ReRAM accelerator architecture to build a novel always-on key-word spotting (KWS) system. The KWS design can switch between different precision modes to capture the relevant sound with high accuracy. The experimental results show the precision-adjustable KWS architecture saves considerable operating energy when fed with realistic test-sets of audio data.

ReRAM-based Computing-in-Memory (CiM) architecture has been considered an ideal solution to neural networks, by conducting in-situ matrix multiplications without moving the neural parameters from memory cells. However, we found that keeping the parameters static in ReRAM cells, i.e. weight-static processing, is not the sole choice to implement emerging graph neural networks (GNNs) that operate on the input of ultra large graphs. Therefore, we propose TARe, a Task-Adaptive CiM architecture that supports multiple different in-situ computing modes for Graph Learning. With the proposed novel hybrid in-situ computing architecture, TARe achieves 451.98× speedup on average over the baseline in SOTA GNN workloads.

Resistive RAM (ReRAM) is a promising device to realize the Computing in Memory (CiM) architecture, suitable for power-constrained IoT systems. Because of low leakage, the dot-production operations in ReRAM crossbars dominate the chip power, especially when implementing low-precision neural networks. This work investigates the correlation between the cell resistance state and the crossbar operation power, and proposes a State-Aware ReRAM Accelerator (SARA) architecture for energy-efficient neural networks. With the proposed state-aware network training and mapping strategy, crossbars in the ReRAM accelerator can perform in a lower-power state. The evaluation shows that our design reduces 47% energy over the baseline without compromising the network accuracy.

Precision-tuning is a popular approach of approximate computing to trade-off excessive computation exactness for power and efficiency gains. Particularly, it has been proved useful to reduce the computation and memory overhead for the deep neural networks on embedded and IoT usage. However, the switching overhead of precision tuning in hardware severely impacts its applicability and effectiveness to save more energy by quickly reacting to the change of environment, user constraint or input quality. This work for the first time investigates the feasibility of agile and cost-free precision tuning for neural network accelerators to benefit from approximate computing. The proposed Processing in Memory (PIM) CNN accelerators fully utilize the normally-off characteristics of memristor crossbars to achieve instant network precision tuning without worrying about the model reloading penalty. The ReRAM-based accelerator, with the proposed neural parameter mapping policy and the novel mixed-model training method, involves negligible precision-switching latency and power consumption compared with traditional variable precision accelerators. The proposed mixed-model training perfectly unifies the neural models of different precision into a single ReRAM array without compromising the accuracy, and the ReRAM accelerator could save 58.3%-62.47% area overhead compared with conventional designs that have to program multiple independent models into ReRAM arrays for precision tuning.