Sparse activation is caused by the activation function (e.g., ReLU) in the feed-forward network (FFN) of large language models (LLMs), and recently emerges as a promising method for LLM inference acceleration in resource-constrained scenarios by effectively reducing computational workload and memory requirements with >80 % predicted dynamic sparsity. In this paper, we identify the heavy and dynamic data transfer is the primary reason for the significant synchronization and poor GPU utilization during decoding phase of LLM inference with sparse activation, and propose to apply the near-data-processing (NDP) architecture to handle the dynamic sparse activation, while addressing three critical challenges for further NDP-GPU collaboration optimization. (1) Under-utilization of DRAM bandwidth during memory access of NDP. (2) Workload imbalance across channels during computation of NDP. (3) Time-consuming parsing of the sparse predicted pattern during NDP-GPU collaboration. To tackle the above challenges, we present SpAct-NDP, the NDPGPU heterogeneous architecture for efficient LLM inference with sparse activation. (1) For the memory access during NDP, we design the the specific sparsity-aware weight mapping strategy considering the characteristics of sparse activation to improve DRAM bandwidth utilization by balancing the bank workload and eliminating redundant memory access. (2) For the computation during NDP, we propose two-level heuristic scheduling system to achieve channel-wise workload balance. (3) For the collaboration of NDP-GPU, we point out that the parsing of the predicted sparse pattern is more suitable for GPUs with high parallelism and propose the request-weight pair parsing mechanism according to the input requests and sparse pattern on GPU, reducing $\sim 3 \times$ execution time and $\sim 9 \times$ memory. Experiments show that SpActNDP achieves up to $2.17 \times$ and $1.92 \times$ end-to-end speedup and $1.53 \times$ and $1.45 \times$ energy efficiency compared with the SOTA software frameworks for LLM with sparse activation on NVIDIA RTX 3090 and NVIDIA Tesla A100.

As an emerging type of AI computing accelerator, SRAM Computing-In-Memory (CIM) accelerators feature high energy efficiency and throughput. However, various CIM designs and under-explored mapping strategies impede the full exploration of compute and storage balancing in SRAM-CIM accelerator, potentially leading to significant performance degradation. To address this issue, we propose CIM-Tuner, an automatic tool for hardware balancing and optimal mapping strategy under area constraint via hardware-mapping co-exploration. It ensures universality across various CIM designs through a matrix abstraction of CIM macros and a generalized accelerator template. For efficient mapping with different hardware configurations, it employs fine-grained two-level strategies comprising accelerator-level scheduling and macro-level tiling. Compared to prior CIM mapping, CIM-Tuner’s extended strategy space achieves 1.58× higher energy efficiency and 2.11× higher throughput. Applied to SOTA CIM accelerators with identical area budget, CIM-Tuner also delivers comparable improvements. The simulation accuracy is silicon-verified and CIM-Tuner tool is open-sourced at https://github.com/champloo2878/CIM-Tuner.git.

Optical computing is becoming a promising solution to meet the growing computational demands of increasingly large-scale deep neural networks (DNNs). However, high power consumption from analog-to-digital (ADC) and digital-to-analog (DAC) conversions poses significant challenges for optical computing. Spiking Neural Networks (SNNs), with their binary spike-based input and output, show the potential to address this issue by reducing the need for high-precision DAC/ADC.In order to exploit the complementary nature of optical computing and spike-based processing, this paper proposes the Partially Incoherent Configurable Optical Computing Architecture for SNN Acceleration (PICoSNN). We address three critical challenges: phase errors in coherent optical computing, limited configurability in weight-stationary architectures, and inefficient mapping of general SNNs to optical computing hardware. We integrate partially incoherent tensor cores with optical leaky integrate-and-fire neurons, minimizing ADC/DAC overhead while supporting dynamic weight mapping. Further, we propose KV Spiking Self-Attention to enable efficient attention with 1-bit multiplications. Experimental results show that PICoSNN achieves up to 70.54× higher throughput and 8.13× lower energy consumption compared to ASIC implementations, while delivering 15.46× better throughput per area and 17.67× better energy efficiency per area than state-of-the-art photonic accelerators.

Large language models (LLMs) are foundational to artificial general intelligence (AGI), while imposing stringent hardware demands in computational power and memory bandwidth. To meet these demands, recent advances in hybrid bonding offer new opportunities through high-bandwidth, low-latency logic-memory 3D integration. Due to the spatial distribution of bandwidth in 3D-stacked DRAM, accesses to remote DRAM banks need to traverse the network-on-chip (NoC). Since GEMM is the dominant operator in LLM workloads, we analyze existing distributed GEMM algorithms and show that they suffer from communication inefficiency on mesh topologies. To address these challenges, we present FlashGEMM, a mesh-aware distributed GEMM algorithm that achieves efficient communication-computation overlap and optimized execution flow for 3D-stacked LLM accelerators. Evaluations show that FlashGEMM delivers up to 1.50× improvement in time-to-first-token (TTFT) and 7.11× improvement in throughput over prior methods.

Reasoning with Large Language Models (LLMs) has become a pivotal research topic because their logical abilities significantly surpass those of standard LLMs. LLM reasoning typically forms multiple chains of thought, action-by-action, and selects the best one as the final answer. However, the inference overhead of LLM reasoning is more than an order of magnitude higher than that of LLM. Despite the emerging shift towards memory-optimized algorithms and near-memory hardware, we still face the following challenges: (1) Existing memory-centric algorithms (e.g., KV cache technique) have low computational utilization (< 4% on NVIDIA A100 GPU) due to intensive memory access for inter-action data. (2) Emerging hardware architectures (e.g., near-memory processing) fail to fully utilize the inherent parallelism due to dependencies among models, leading to low utilization of memory bandwidth.To tackle these challenges, we propose Endor, a hardware-algorithm co-design to accelerate the inference of LLM reasoning efficiently. We identify that the auto-regressive decoding of LLM reasoning changes from the token level to the action level in terms of the computing paradigm. At the algorithm level, we propose a "nearly-decode-only" method which encompasses an efficient inter-action cache reuse method and a prediction-based pipeline optimization to reduce computation overhead. At the hardware level, we propose Endor-NMP, a near-memory accelerator featuring a score-aware cache management architecture and a heterogeneous mapping dataflow. Endor fully exploits both interaction and intra-action parallelism to improve memory bandwidth utilization. Experimental results demonstrate that neither existing algorithms nor hardware can achieve the expected acceleration. Endor achieves an end-to-end average speedup of 2.97× and 2.52× compared to the NVIDIA A100 GPU and advanced LLM accelerators on multiple models and datasets.

Generative models have achieved remarkable success across various applications, driving the demand for multi-GPU computing. Inter-GPU communication becomes a bottleneck in multi-GPU computing systems, particularly on consumer-grade GPUs. By exploiting concurrent hardware execution, overlapping computation and communication latency becomes an effective technique for mitigating the communication overhead. We identify that an efficient and adaptable overlapping design should satisfy (1) tile-wise overlapping to maximize the overlapping opportunity, (2) interference-free computation to maintain the original computational performance, and (3) communication agnosticism to reduce the development burden against varying communication primitives. Nevertheless, current designs fail to simultaneously optimize for all of those features.

The rapid scaling of large language models (LLMs) has significantly increased GPU memory pressure, which is further aggravated by training optimization techniques such as virtual pipeline and recomputation that disrupt tensor lifespans and introduce considerable memory fragmentation. Such fragmentation stems from the use of online GPU memory allocators in popular deep learning frameworks like PyTorch, which disregard tensor lifespans. As a result, this inefficiency can waste as much as 43% of memory and trigger out-of-memory errors, undermining the effectiveness of optimization methods.

Deep neural network (DNN) accelerators integrated with RISC-V Instruction Set Architecture (ISA) extensions have enabled efficient computing on resource-constrained platforms. However, their specialization in regular compute patterns limits effectiveness on irregular workloads, making it challenging to achieve high throughput and energy efficiency. To tackle these challenges, we present ReNN-RV, which integrates a computation-aware RISC-V ISA extension with an instructiondriven processing pipeline to efficiently accelerate run-time reconfigurable processing elements (RePEs). The computation-aware ISA employs configurable opcodes and custom encodings to support fine-grained task scheduling, while an instructiondriven pipeline implements it with minimal control complexity. Moreover, theRePEaccelerator provides seamless switching between multiply-accumulate (MAC) and non-MAC operations by configuring a path multiplexer to realize multiple operators at run time. Experimental results demonstrate that ReNN-RV achieves average reductions of 14.6× in cycle count and 15.3× in execution time across representative DNN workloads compared with the baseline RISC-V design. On average, ReNN-RV outperforms state-of-the-art designs by 10.1× for energy efficiency and 10.3× for computational throughput.

Neural networks (NNs) have exhibited excellent performance in various fields of artificial intelligence. However, the primary operations in these mainstream models, including matrix-vector multiplication (MVM), element-wise multiplication (EWM), and depth-wise convolution (DWConv), require massive data movements during computation, which greatly impacts NNs’ inference performance. The emerging Processing-In-Memory (PIM) architectures have shown great potential to overcome the memory wall problem. However, constrained by the supported data format and operator type, directly adopting PIM architectures for neural network acceleration faces three challenges: (1) Floating-point (FP) format has been widely adopted for ensuring high algorithm accuracy. However, Resistive Random-Access Memory (RRAM)-based analog PIM architectures perform integer (INT) MVMs in the analog domain, limiting their application to the more accurate FP format; (2) Static Random-Access Memory (SRAM)-based digital PIM architectures require additional circuits to support the FP format, and the SRAM capacity cannot satisfy the storage requirement of latest large language models (LLMs); (3) When performing the operators with few accumulation steps, such as EWMs and DWConvs, only few memory units in PIM architecture are activated, resulting in severe device under-utilization. To tackle the above challenges, this article proposes an RRAM and 3D-SRAM-based hybrid PIM architecture, achieving FP-based algorithm accuracy, high device utilization, and high energy efficiency. At the software level , we first analyze the impact of quantization errors on NN’s inference accuracy. For the quantization error-insensitive MVM operations, we propose the PIM-oriented exponent-free non-uniform (PN) data format. The proposed PN format can be flexibly adjusted to fit the non-uniform distribution and approach FP-based algorithm accuracy using bit-slicing-based full INT operations. For the quantization error-sensitive EWM/DWConv operations, we introduce the multiplication-free approximated FP multiplications to reduce the additional hardware overhead. At the hardware level , we propose a hybrid PIM architecture, including an RRAM analog PIM using shift-and-add for PN-based MVMs, and a 3D-SRAM digital PIM with high utilization for DWConv/EWM operations. Extensive experiments on CNNs and attention-free LLMs validate that the proposed PIM architecture achieves up to 99.4× and 33.9× speedup with 5697.7× and 8.2× energy efficiency improvement compared to GPU and PIM-baseline, respectively. With the proposed PN format and approximated FP multiplications, the algorithm accuracy of CNNs and attention-free LLMs can be improved by up to 3.01% and 10.18%, respectively.

Large Language Models (LLMs) with 70 billion or more parameters are increasingly being deployed in cloud-based Model-as-a-Service (MaaS) scenarios. To meet the demands of such deployments, MaaS providers require batched LLM decoding systems that can deliver high System Throughput (STP) while minimizing Total Cost of Ownership (TCO). However, existing FPGA-based solutions predominantly focus on small-batch or single-batch inference, which fails to meet the computational requirements of batched LLM decoding, resulting in performance gaps of up to 7.96 \(\times\) . Moreover, the low utilization of multi-head attention operations in batched decoding scenarios, e.g., only 3.72% on A100 GPUs, further constrains throughput and inflates TCO. To address these challenges, this article introduces CD-LLM , a heterogeneous multi-FPGA system designed for efficient batched decoding of LLMs with 70B+ parameters, built upon a C ompute- D edicated architecture. First, we propose a memory-aligned mixed-precision quantization engine to reduce workload. By employing importance-aware quantization, we compress Llama-3.1-70B to an effective 3.45-bit representation and achieve 72.33% bandwidth utilization through memory-aligned data packing. Second, we present a compute-dedicated FPGA architecture that maximizes peak performance by leveraging FPGA-specific resources such as DSPs, BRAMs, and LUTs. The compute-dedicated architecture enables CD-LLM to reach a peak performance of 59.90 TOPS at 600 MHz on U250 FPGA. At last, we introduce a heterogeneous master-slave multi-FPGA system to achieve higher utilization. By pipelining attention and linear layer computations across master and slave FPGAs, CD-LLM achieves utilization rates of 83.08% for linear layers and 68.30% for attention layers. CD-LLM is designed with a heterogeneous multi-FPGA architecture, with an HBM-enabled FPGA as the master accelerator and eight DDR-based FPGAs as slave accelerators. When deployed for inference on the Llama-3.1-70B model with a batch size of 256, CD-LLM achieves a throughput of 2,721.79 tokens/s. This represents a 6.11 \(\times\) improvement in STP and a 4.71 \(\times\) reduction in TCO compared to an eight-card RTX3090 GPU system. Furthermore, CD-LLM substantially outperforms the state-of-the-art eight-card FPGA accelerator FlightLLM, delivering 16.15 \(\times\) higher STP and 14.56 \(\times\) lower TCO.

Transformer-based video generation models have demonstrated significant potential in content creation. However, the current state-of-the-art model employing “ 3 D full attention” encounters substantial computation and storage challenges. For instance, the attention map size for $\operatorname{Cog}$ VideoX-5B requires 56.50 GB, and generating a video of 49 frames takes approximately 1 minute on an NVIDIA A100 GPU under FP16. Although model quantization has proven effective in reducing both memory and computational costs, applying it to video generation models still faces challenges in preserving algorithm performance while ensuring efficient hardware processing. To address these issues, we introduce PARO, a video generation accelerator with patternaware reorder-based attention quantization. PARO investigates the diverse attention patterns of 3D full attention and proposes a novel reorder technique to unify these patterns into a unified “block diagonal” structure. Block-wise mixed precision quantization is further applied to achieve lossless compression under an average bitwidth of 4.80 bits. In terms of hardware, to overcome the limitation of existing mixed-precision computing units could not fully utilize the attention map bitwidth to accelerate $Q K$ multiplication, PARO designs an output-bitwidth aware mixedprecision processing element (PE) array through hardwaresoftware co-design. This approach ensures that the mixedprecision characteristics are fully utilized to enhance hardware efficiency in the bottleneck attention computation. Experiments demonstrate that PARO delivers up to $2.71 \times$ improvement in end-to-end performance compared to an NVIDIA A100 GPU and achieves up to $6.38 \sim 7.05 \times$ speedup over state-of-the-art ASICbased accelerators on the CogVideoX-2B and 5B models.

Network-on-Chip (NoC) accelerators with heterogeneous Processing-in-Memory (PIM) cores achieve superior performance than homogeneous ones for neural networks. Dedicated simulators and architecture search frameworks are pivotal for obtaining performance, power, and area (PPA) metrics, as well as guiding the design process. However, existing simulators are primarily designed for homogeneous NoC and lack support for simulating heterogeneous PIM-based NoC architectures. Besides, current search frameworks for heterogeneous NoC architectures only focus on workload allocation and mapping strategies, failing to explore heterogeneous PIM configurations in a larger design space. In this work, we propose HPIM-NoC, a joint simulation and search framework for heterogeneous PIM-based NoC architectures. HPIM-NoC not only supports the simulation of heterogeneous PIM cores, but also provides more accurate latency results by introducing NoC transmission delays and pipelines in co-simulation. HPIM-NoC implements a three-stage heterogeneous search process based on priori knowledge and employs a specific simulated annealing algorithm tailored for heterogeneous architecture search. The search process is accelerated by precomputing core PPA metrics and reducing NoC simulation frequency. In addition, the framework integrates a customized layout algorithm to optimize the placement of heterogeneous NoC, minimizing communication latency and overall area. Experimental results on various neural networks demonstrate that HPIM-NoC can quickly find near-optimal configurations within a limited time. The proposed acceleration method reduces the search time of HPIM-NoC by $2.12 \times$, $2.17 \times$, and $2.96 \times$, respectively. Compared to homogeneous architectures, the Fusions of Metrics (FoMs) of heterogeneous PIM-based NoC architectures found by HPIM-NoC are reduced by $\mathbf{1. 1 8 \%, ~} \mathbf{1 6. 9 4 \%}$, and $\mathbf{3 7. 4 1 \%}$ for ResNet-18 under three settings, respectively.

For large language model (LLM) acceleration, FPGAs face two challenges: insufficient peak computing performance and unacceptable accuracy loss of model compression. This paper proposes FMC-LLM to enable FPGAs for efficient batched decoding of 70B+ LLMs.

The computational and memory demands of Deep Learning (DL) models, from convolutional neural networks to Large Language Models (LLMs), are experiencing a notable surge. The sparsification (e.g., weight pruning and sparse attention) represents a significant approach to reducing latency and energy consumption. However, it is non-trivial to identify a good trade-off between model accuracy and hardware efficiency. Existing work has sought to mitigate the hardware complexity overhead through structured sparsity, yet the resulting accuracy loss remains considerable (e.g., more than 6% accuracy drop with 50% structured sparsity on OPT-6.7B and Llama2-7B).To address the above challenges, this paper proposes Transposable Block-wise Structured Sparsity (TBS). Our key insight is that the weight matrices of the forward and backward pass are transposed to each other during DL training. Exploiting this transposition property facilitates obtaining a structured sparsity pattern that is closer to the unstructured sparsity. In contrast, existing studies explore only one-dimensional structured sparsity. In light of these observations, we propose the transposable block-wise structured sparsity pattern with an efficient end-to-end sparse training method. This method improves accuracy by up to 2.58% over other structured sparsity studies under the same sparsity degree. At the micro-architecture level, we propose TB-STC, a Transposable Block-wise N:M Sparse Tensor Core to efficiently and flexibly facilitate the TBS pattern. TB-STC introduces an adaptive codec architecture for on-the-fly storage format conversion with a higher bandwidth utilization (1.47 ×), and implements an I/O-aware configurable architecture for sparsity-aware scheduling with a better computational utilization (1.57×). Compared with existing work, TB-STC improves the Energy-Delay Product (EDP) by an average of 3.82 × and offers an enhanced accuracy-EDP Pareto frontier across various sparse DL models.

Near DRAM Processing (NDP) architectures have emerged to be a promising solution for commercializing in-memory computing and addressing the “memory wall” problem, especially for the memory-intensive machine learning (ML) workloads. In NDP architectures, the Processing Units (PUs) are distributed next to different memory units to exploit the high internal bandwidth. Therefore, in order to fully utilize the bandwidth advantage of NDP architectures for ML applications, meticulous evaluations and optimizations of data placement in DRAM and workload scheduling among different PUs are required. However, existing simulation and compilation tools face two insuperable obstacles to achieving these targets. On the one hand, tools for traditional von Neumann architectures only focus on the data access behaviors between the host and DRAM and treat DRAM as a whole part, which cannot support NDP architectures with multiple independent processing and memory units working simultaneously. On the other hand, existing NDP simulators and compilers are designed for specific DRAM technology and NDP architecture, lacking compatibility for various NDP architectures. In order to overcome these challenges and optimize data mapping and workload scheduling for different NDP architectures, we propose UniNDP, a unified NDP compilation and simulation tool for ML applications. Firstly, we propose a unified tree-based NDP hardware abstraction and the corresponding instruction set, enabling the support for various NDP architectures based on different DRAM technologies. Secondly, we design a cycle-accurate and instruction-driven NDP simulator to evaluate hardware performance by accurately tracking the working status of memory elements and PUs. The accurate simulation can provide effective guidance for compilation. Thirdly, we design an NDP compiler that optimizes data partition, mapping, and workload scheduling in different DRAM hierarchies. Furthermore, to enhance the compilation efficiency, we propose a hardware status-guided search space pruning strategy and a fast performance predictor using DRAM timing parameters. Extensive experimental results show that, compared to existing mapping and compilation methods, UniNDP can achieve 1.05-3.43 $\times$ speedup across multiple NDP architectures and different ML workloads. Furthermore, based on the results of UniNDP, we provide insights for the future NDP architecture design and deployment in ML applications.

The emerging analog Processing-in-Memory (PIM) architectures have shown great potential to overcome the memory wall problem and accelerate neural network (NN) inference. However, different from digital architectures, the computation accuracy of analog PIM architectures is directly impacted by various errors, which are related to both software and hardware parameters. Existing PIM simulators mainly adopt the bit-and-crossbar slicing paradigm to evaluate the accuracy under various errors. Each MVM operation is performed bit by bit and crossbar by crossbar, which is extremely time-consuming, especially for models with a larger number of parameters, such as large language models (LLMs).In this work, we propose an error-injected robustness metric, unifying various errors into the weight dimension and facilitating joint error analysis. Based on the error-injected robustness metric, we propose a Non-Ideal PIM Accuracy (NIPA) evaluation model for relative accuracy evaluation, considering the coupling effect (i.e., various errors can be affected by the same factor) among various errors using NN’s prior information. We further propose a non-slicing absolute accuracy evaluation method, eliminating the need for the time-consuming bit-and-crossbar slicing process. Extensive experiments on CNNs and LLMs validate that the proposed NIPA evaluation model achieves high correlations of up to 0.91 with the absolute accuracy evaluated by DNN+NeuroSim. At the same time, compared to existing bit-and-crossbar slicing evaluation methods, the proposed non-slicing absolute accuracy evaluation method achieves up to 105.8× speedup with average evaluation errors as low as 0.29%.

3D Gaussian Splatting (3DGS) has emerged as a promising approach for high-fidelity scene reconstruction and has been widely adopted in Simultaneous Localization and Mapping (SLAM) systems.3DGS SLAM requires incremental training and rendering of Gaussians in real-time from continuous camera viewpoints.To match the streaming nature of SLAM, 3DGS-based mapping must sustain over 30 frames per second (FPS), which is a widely recognized threshold for maintaining accurate tracking and mapping quality.Existing GPU-based solutions and prior accelerators fall short of this target, primarily due to redundant training computation, unnecessary loss computing, and irregular memory access patterns.To address these challenges, we propose REACT3D, a real-time edge accelerator designed for incremental training in 3DGS SLAM systems.At the algorithmic level, we introduce spatial consistency and convergence aware sparsification, which eliminates redundant computation in both forward and backward rendering by predicting under-optimized regions based on spatial coherence and convergence dynamics.At the architectural level, we design a pixel blockwise fine-grained dataflow to eliminate explicit loss computing, establish a tightly coupled pipeline, and improve hardware utilization.Furthermore, we develop a Content Addressable Memory (CAM)-based Dual-index Gaussian Buffer to resolve discontinuous * Equal contribution.

SRAM computing-in-memory (CIM) accelerators have evolved as a promising solution to the memory wall problem in neural network (NN) models. By integrating memory and compute resources in each macro, CIM accelerators offer massive in-situ computing parallelism and large memory capacity, enabling spatial mapping with layer fusion and potentially keeping layers stationary in CIM. However, CIM’s memory-compute coupling (MCC) feature poses challenges in designing CIM accelerators. From an architecture aspect, designers must balance CIM’s memory and compute resources by optimizing the macro’s memory-compute ratio (MCR) configuration across diverse scenarios. From a mapping aspect, conventional mappings, which allocate each macro exclusively to each layer, face two major problems: a layer-fusion dilemma (the accelerator suffers from excessive memory access due to layer replications or performance degradation due to load imbalance) and a layer-eviction issue (storing layers stationary in CIM is usually infeasible due to limited CIM capacity). To address these challenges, this paper introduces MCC-DSE, an MCC-aware Design Space Exploration framework for architecture-mapping co-optimization of CIM accelerators. We also propose a three-axis CIM division mapping, which interleaves multiple layers in each macro to concurrently optimize memory access and performance during layer fusion as well as reserves a part of CIM memory in each macro for layer pinning. Compared to baseline architecture and mapping, MCC-DSE shows a 1.4x 8.3x EDP reduction across various workloads and chip areas. Moreover, MCC-DSE provides insights into CIM accelerator optimization, such as selecting optimal MCR and configuring CIM dynamically for different scenarios.

Hybrid convolutional neural network (CNN) and Transformer networks are emerging in computer vision, combining convolutional, linear, and attention layers to achieve high accuracies with moderate model sizes. Developing the accelerators for hybrid networks is pivotal to simultaneously optimize the static matrix multiplication (MM) in convolutional and linear layers, as well as dynamic MM in attention layers. However, the existing accelerators are primarily designed for either CNNs or Transformers, resulting in increased data movement to support dynamic MM and potential under-utilization of hardware for static MM. To enhance computational performance and energy efficiency for hybrid networks, we propose HyCTor, an accelerator featuring flexible output-stationary (OS) and weight-stationary (WS) dataflows, along with a multicore extension for higher throughput. The parallel array of HyCTor supports interlayer slicing and intralayer splicing to improve the utilization for static MM, and enables seamless switching between OS and WS dataflow to minimize the data movement in dynamic MM. By leveraging structured sparsity in OS dataflow and unstructured sparsity in WS dataflow, the computational efficiency is further boosted for each layer through flexible dataflow selection based on the sparsity ratio. Besides, a novel QuadLoop-mesh topology is proposed to address the complex data dependencies in hybrid networks and minimize data transmission distances in the multicore HyCTor. Experimental results on ResNet-18, ViT-B, and TransIAR-AF show that the proposed single-core HyCTor achieves$1.83\times $,$1.65\times $, and$2.41\times $speedup than state-of-the-art (SOTA) accelerators with 100% utilization rate in most layers, and$3.82\times $–$38.5\times $speedup than RTX4090 GPU. The energy efficiency of HyCTor is improved by$1.81\times $–$8.77\times $compared with SOTA accelerators. Moreover, the 4-core HyCTor achieves speedups of$3.32\times $,$2.58\times $, and$2.91\times $, while the 16-core HyCTor achieves speedups of$7.05\times $,$4.05\times $, and$9.64\times $compared to 1-core HyCTor on three networks.

Recently, sparse tensor algebra (SpTA) plays an increasingly important role in machine learning. However, due to the unstructured sparsity of SpTA, the general-purpose processors (e.g., GPU and CPU) are inefficient because of the underutilized hardware resources. Sparse kernel accelerators are optimized for specific tasks. However, their dedicated processing units and data paths cannot effectively support other SpTA tasks with different dataflow and various sparsity, resulting in performance degradation. This paper proposes FEASTA, a Flexible and Efficient Accelerator for Sparse Tensor Algebra. To process general SpTA tasks with various sparsity efficiently, we design FEASTA meticulously from three levels. At the dataflow abstraction level, we apply the Einstein Summation on the sparse fiber tree data structure to model the unified execution flow of general SpTA as joining and merging the fiber tree. At the instruction set architecture (ISA) level, a general SpTA ISA is proposed based on the execution flow. It includes different types of instructions for dense and sparse data, achieving flexibility and efficiency at the instruction level. At the architecture level, an instruction-driven architecture consisting of configurable and high-performance function units is designed, supporting the flexible and efficient ISA. Evaluations show that FEASTA has 5.40× geomean energy efficiency improvements compared to GPU among various workloads. FEASTA delivers 1.47× and 3.19× higher performance on sparse matrix multiplication kernels compared to state-of-the-art sparse matrix accelerator and CPU extension. Across diverse kernels, FEASTA achieves 1.69-12.70× energy efficiency over existing architectures.

The utilization of large-scale neural networks on Processing-In-Memory (PIM) accelerators encounters challenges due to constrained on-chip memory capacity. To tackle this issue, current works explore model compression algorithms to reduce the size of Convolutional Neural Networks (CNNs). Most of these algorithms either aim to represent neural operators with reduced-size parameters (e.g., quantization) or search for the best combinations of neural operators (e.g., neural architecture search). Designing neural operators to align with PIM accelerators' specifications is an area that warrants further study. In this paper, we introduce the Epitome, a lightweight neural operator offering convolution-like functionality, to craft memory-efficient CNN operators for PIM accelerators (EPIM). On the software side, we evaluate epitomes' latency and energy on PIM accelerators and introduce a PIM-aware layer-wise design method to enhance their hardware efficiency. We apply epitome-aware quantization to further reduce the size of epitomes. On the hardware side, we modify the datapath of current PIM accelerators to accommodate epitomes and implement a feature map reuse technique to reduce computation cost. Experimental results reveal that our 3-bit quantized EPIM-ResNet50 attains 71.59% top-1 accuracy on ImageNet, reducing crossbar areas by 30.65X. EPIM surpasses the state-of-the-art pruning methods on PIM.

Sparse Matrix-Matrix Multiplication (SpMM) is one of the key operators in many fields, showing dynamic features in terms of sparsity, element distribution, and data dependency. Previous studies have proposed FPGA-based SpMM accelerators with fixed configurations of on-chip dataflow, leaving three major challenges unsolved: 1) Partitioning matrices with the fixed sub-matrix size to fit limited on-chip buffer on FPGA leads to performance loss because the optimal sub-matrix size to minimize memory access varies with dynamic sparsity. 2) The fixed row-wise allocation scheme of sparse elements in streaming architecture leads to unbalanced workloads because of dynamic element distribution across sparse matrix rows. 3) Read-after-write (RAW) hazard caused by floating-point adder makes the elements in one row cannot be processed consecutively. Architectures with fixed execution order rely on time-consuming pre-processing to deal with dynamic data dependency. Motivated by the observation that fixed configurations lead to performance loss, we propose DySpMM by introducing the dynamic design methodology to SpMM architectures. The configurable data distributor is introduced to enable dynamic sub-matrix size, achieving up to 3.79× less memory access amount. The element-wise allocator is designed for dynamic workload balance, improving utilization up to 3.74×. The interleaved reorder unit is proposed to reorder the elements and dynamically avoid RAW hazards at runtime, avoiding time-consuming pre-processing. We implement DySpMM on U280 FPGA, and the evaluation shows that it achieves 1.42× geomean throughput compared with the state-of-the-art accelerator Sextans and 1.78× energy efficiency compared with V100S GPU.

Dynamic neural network is an emerging research topic in deep learning. Dynamic networks selectively skip redundant computations conditioned on the input during inference (i.e., dynamic inference). And they have demonstrated superior trade-offs between accuracy and inference efficiency. However, memory I/O turns irregular and dominant because of the fine-grained computation skip in dynamic networks. Processing-In-Memory (PIM) can perform Matrix-Vector Multiplications inside the memory, eliminating the data movement of network parameters. So, it is promising to address the memory I/O challenge. However, deploying dynamic networks on PIM architectures faces severe performance degradation caused by (1) Pipeline stall when deciding on computation to be skipped. (2) Mismatch between fine-grained algorithm computation skip and coarse-grained hardware computing granularity. (3) Improper proxy of hardware performance during training. To tackle these problems, we propose DyPIM, the dynamic inference-enabled PIM accelerator with software-hardware co-optimizations. At the algorithm level, a PIM-friendly dynamic network with a standalone mask generation network and a throughput-optimal training technique is proposed. At the hardware level, a PIM architecture supporting dynamic networks is proposed, with a pipeline controller to process the dynamic dataflow. Peripheral circuits are also designed in processing units to enable non-contiguous activating of non-zero wordlines to better utilize the computation skip. Experiments show that DyPIM can achieve 1.52x to 2.74x speedup and 2.05x to 3.95x throughput improvement over the existing PIM architectures for Res Net networks.

Attention-free large language models (LLMs), such as Mamba and RWKV, have emerged as promising architectures to address the quadratic attention complexity of Transformer models. The inference bottleneck of these models lies in memory-bound matrix-vector multiplications (MVMs) and element-wise multiplications (EWMs). The emerging RRAM/SRAM-based Processing-In-Memory (PIM) architectures have shown great potential to overcome the memory wall problem. However, constrained by the supported data format and operator type, directly adopting PIM architectures for attention-free models faces three challenges: (1) RRAM-based analog PIM architectures perform integer (INT) MVMs using voltage, current, and conductance in the analog domain, limiting their application to the more accurate floating point (FP) data format; (2) SRAM-based digital PIM architectures require additional decoder circuits to support FP format, and the SRAM capacity cannot satisfy the storage requirement of LLMs; (3) When performing EWMs using PIM architectures, only one row/column or the diagonal memory cells are activated, resulting in severe device under-utilization.

Recent advances in hardware/algorithm co-design for spiking neural networks have demonstrated its potential for jointly optimizing algorithmic performance while minimizing hardware overhead. However, the gigantic mixed-variable hard-ware/algorithm co-design space and time-consuming hardware verification still pose an intractable challenge for solutions exploration. To tackle these problems, 1) we propose a generic three-phase hardware/algorithm co-design framework. In this framework, 2) we target a reconfigurable neuromorphic processor, and parameterize the hardware and network architecture in a unified design space. 3) We propose a generic analytical model to estimate the parameter size and power consumption, which can support fast candidate evaluation during the exploration. 4) We extend vanilla TPE (a single-objective optimization algorithm) to MOTPE/D, a generic Multi-objective optimization (MOO) algorithm, by introducing a decomposition strategy.

The execution of deep neural network (DNN) algorithms suffers from significant bottlenecks due to the separation of the processing and memory units in traditional computer systems. Emerging memristive computing systems introduce an in situ approach that overcomes this bottleneck. The nonvolatility of memristive devices, however, may expose the DNN weights stored in memristive crossbars to potential theft attacks. Therefore, this article proposes a 2-D permutation-based protection (TDPP) method that thwarts such attacks. We first introduce the underlying concept that motivates the TDPP method: permuting both the rows and columns of the DNN weight matrices. This contrasts with previous methods, which focused solely on permuting a single dimension of the weight matrices, either the rows or columns. While it is possible for an adversary to access the matrix values, the original arrangement of rows and columns in the matrices remains concealed. As a result, the extracted DNN model from the accessed matrix values would fail to operate correctly. We consider two different memristive computing systems (designed for layer-by-layer and layer-parallel processing, respectively), and demonstrate the design of the TDPP method that could be embedded into the two systems. Finally, we present a security analysis. Our experiments demonstrate that TDPP can achieve comparable effectiveness to prior approaches, with a high level of security when appropriately parameterized. In addition, TDPP is more scalable than previous methods and results in reduced area and power overheads. The area and power are reduced by, respectively,$1218\times $and$2815\times $for the layer-by-layer system and by$178\times $and$203\times $for the layer-parallel system compared to prior works.

Object detection via deep neural networks has undergone considerable advancements in recent years. Yet, the detection of smaller objects, specifically those with a few pixels (i.e.,2pixels), is still challenging compared with large objects (i.e., > 962pixels). Existing methods commonly apply high-resolution features or complex super-resolution strategies based on the two-stage Faster Region Convolutional Neural Network (RCNN). They sequentially apply localization and classification stages after a shared feature map extracted by one single backbone network. However, these methods cause low detection accuracy of small objects, high computational overhead, and waste of hardware resources. In this paper, we develop a high-accuracy and real-time small object detection system with negligible computational overhead and low hardware idleness. At the software level, we propose a two-stage Coarse-to-Fine Decoupling RCNN (CFD RCNN) with three techniques: (1) The shared backbone decoupling for localization and classification to achieve high accuracy for both tasks; (2) The training method using backbone feature upsampling for localization with low computational overhead; (3) The object cropping strategy from the original high-resolution image for high-accuracy classification. At the hardware level, we propose a virtualized FPGA accelerator with the Dynamic Resource Allocation (DRA) strategy. The DRA strategy reallocates the hardware resources, considering the workload and resource preference of each stage in CFD RCNN to reduce hardware idleness. Extensive experiments on the TT100K and GTSDB datasets using Xilinx ZCU102 FPGA show that the proposed small object detection system can achieve 2.9% improvement in mean average precision (mAP) compared with state-of-the-art (SOTA) algorithms and raised the throughput from 18.9 FPS to > 26.0 FPS (~1.37×) compared with existing accelerators.

Single Photon Avalanche Diode (SPAD)-based direct time-of-flight (dToF) depth sensors are widely used in Internet of Things (IoT) devices due to their high accuracy. Existing SPAD-based dToF sensors measure depth by continually accumulating the depth-measured value in a histogram. However, histogram-based methods typically have low convergence speed (~10 frames per second (FPS)) and large memory overhead (MB-level), hindering their use in real-time embedded IoT devices. To overcome these two challenges, we propose SSC, a histogram-free Spatial and Statistical Correlation based depth measurement method. On the one hand, SSC applies the spatial correlation of the adjacent pixels to accelerate the convergence speed. On the other hand, SSC explores the statistical correlation of depth measurements to reduce the memory overhead. In order to implement SSC with small hardware area and low power, we design mert-dToF, a memory-efficient and real-time dToF sensor for efficient execution. mert-dToF abstracts mainly operations in SSC into four basic operators and designs corresponding hardware with a fine-grained pipeline to maximize resource reuse and computational parallelism. Extensive experiments show that compared with state-of-the-art (SOTA) histogram-based dToF sensors, mert-dToF achieves ~8% accuracy improvement and 7.80× speedup (from 6.24 FPS to 48.70 FPS). The memory overhead is reduced by up to 60.91% (from 48 KB to 18.75 KB).

Graph-based approximate nearest neighbor search (ANNS) algorithms achieve the best accuracy for fast high-recall searches on billion-scale datasets. Because of the irregular and large-volume data access, existing CPU-based systems suffer from heavy data movements when dealing with graph-based ANNS algorithms. Near-memory-computing (NMC) architectures have demonstrated great potential in boosting the performance of big-data processing. However, existing NMC architectures face two serious problems when processing graph-based ANNS algorithms: (1) the memory capacity of main memory level NMC (e.g., 64GB) cannot meet the storage requirement of ANNS on billion-scale datasets (e.g., 800GB), resulting in heavy data transfers between main memory and storage; (2) the contradiction between the irregular and fine-grained graph access and the page-level read granularity hinder the throughput of storage level NMC.This paper proposes Pyramid, the processing-in-hierarchical-memory architecture for graph-based ANNS on billion-scale datasets. Pyramid combines the internal bandwidth benefits of main memory level NMC with the capacity benefits of storage level NMC. A hierarchical graph-cluster-based ANNS is also proposed for Pyramid. It transforms the irregular data access on large-scale graphs into the irregular access on small-scale graphs at the main memory level and regular sequential in-cluster access at the storage level. Experimental results show that with the same recall of 0.9, Pyramid improves the throughput by 21.1~72.8× and 26.0~50.7× compared with existing CPU/GPU-based ANNS systems on million-scale and billion-scale datasets, respectively.

Processing-in-memory (PIM) architectures have shown great abilities for neural network (NN) acceleration on edge devices that demand low latency under severe area constraints. Heterogeneous PIM architectures with different PIM implementation approaches such as RRAM-based PIM and SRAM-based PIM can further improve the performance. However, the automatic generation of heterogeneous PIM architectures faces the following two unresolved problems. First, existing work has not considered the design for heterogeneous PIM-based NN accelerators with multiple memory technologies. Second, for PIM with insufficient memory on edge devices, it is challenging to find the optimal runtime weight scheduling strategy in an O(L!) optimization space for the NN with L layers.In this paper, we propose PIM-HLS, an automatic hardware generation tool for heterogeneous PIM-based NN accelerators. Aiming at the problems above, we first point out that heterogeneous PIM can improve the performance under severe area constraints. Then we optimize the architectures for each NN layer by taking the advantage of different memory technologies. We also define the optimization problem of runtime weight scheduling and mapping for the first time, and propose a dynamic-programming-based weight scheduling algorithm to reduce the optimization space to O(L2). We implement PIM-HLS to automatically generate the hardware code and the instructions. Results show that we achieve an averagely 5.9× speedup with 72.8% less area compared with state-of-the-art PIM designs.

Triangle counting (TC) is one of the most fundamental graph analysis tools with a wide range of applications. Modern triangle counting algorithms traverse the graph and perform set intersections of neighbor sets to find triangles. However, existing triangle counting approaches suffer from the heavy off-chip memory access and set intersection overhead. Thus, we propose CLAP, the first content addressable memory (CAM) based triangle counting architecture with the software and hardware co-optimizations. To reduce off-chip memory access and the number of set intersections, we propose the first force-based node index reorder method. It simultaneously optimizes both data locality and the computation amount. Compared with random node indices, the reorder method reduces the off-chip memory access and the set intersections by 61% and 64%, respectively, while providing$\mathbf{2.19}\times$end-to-end speedup. To improve the set intersection parallelism, we propose the first CAM-based triangle counting architecture under chip area constraints. We enable the high parallel set intersection by translating it into content search on CAM with full parallelism. Thus, the time complexity of the set intersection reduces from$O(m+n)$or$O(n\log m)$to$O(n)$. Extensive experiments on real-world graphs show that CLAP achieves$\mathbf{39}\times, \mathbf{27}\times$, and$\mathbf{78}\times$speedup over state-of-the-art CPU, GPU, and processing-in-memory baselines, respectively. The software code is available at: https://github.com/thu-nics/CLAP-triangle-counting

Deep Neural Networks (DNNs) have made significant breakthroughs in various fields. However, their enormous computations and parameters seriously hinder their applications. Emerging Processing-In-Memory (PIM) architectures provide extremely high energy efficiency to accelerate DNN computing. Moreover, Network-on-Chip (NoC) based PIM architectures significantly improve the scalability of PIM architectures. However, the contradiction between high communication and limited NoC bandwidth introduces severe communication conflicts. Existing work neglects the impact of communication conflicts. On the one hand, neglecting communication conflicts leads to the lack of precise performance estimations in the mapping process, making it hard to find optimal results. On the other hand, communication conflicts cause low NoC bandwidth utilization in the schedule process. And there is over 70% latency gap in existing work caused by communication conflicts. This paper proposes communication conflict optimized mapping and schedule strategies for NoC-based PIM architectures. The proposed mapping strategy constructs communication conflict graphs to model communication conflicts. Based on this constructed graph, we adopt a Graph Neural Network (GNN) as a precise performance estimator. Our schedule strategy predefines the communication priority and NoC communication behavior tables for target DNN workloads. In this way, it can improve the NoC bandwidth utilization effectively. Compared with existing work, for typical classification DNNs on the CIFAR and ImageNet datasets, the proposed strategies reduce 78% latency and improve the throughput by 3.33× on average with negligible deployment and hardware overhead. Experimental results also show that our strategies decrease the average gap to ideal cases without communication conflicts from 80.7% and 70% to 12.3% and 1.26% for latency and throughput, respectively.

Deep convolutional neural networks (CNNs) are widely used in Artificial Intelligence of Things (AIoT) systems. Limited by power and area, conventional edge devices are insufficient to handle the cost of CNN computation. The idea of SRAM based Processing-In-Memory (SRAM-PIM) has been advocated to implement CNN on edge devices because of its high area and power efficiency. To further excavate the potential of SRAM-PIM on edge inferences, this paper proposes an SRAM-PIM design with Kullback-Leibler (KL) divergence-based dynamic precision quantization. The proposed quantization method decouples the effect of different CNN layers on accuracy and introduces the SRAM-PIM hardware performance in quantization, realizing SRAM-PIM-aware layer-wise precision adjustment. The proposed SRAM-PIM design has been applied in image classification tasks on edge devices. Our evaluation shows that the implemented design achieves up to 2.03x energy efficiency improvement and 2.54% accuracy improvement compared with existing dynamic precision PIM design. Compared with existing reinforcement-learning-based dynamic quantization method that requires several hours quantization time, the proposed dynamic precision quantization method takes only 26.28us to get the optimal quantization results.

Phase-change memory (PCM) and resistive memory (RRAM) are promising alternatives to traditional memory technologies. However, both PCM and RRAM suffer from limited write endurance. Wear-leveling (WL) techniques are essential to extend the lifetime of these memories before experiencing endurance faults. Beyond the additional usage afforded by WL, row-sparing and focused error correction can extend the lifetime further after wear faults appear. Unfortunately, the need for extended WL techniques continues to become more pressing as scaling exacerbates process variation. Similarly, scaling causes challenges such as more severe noise and crosstalk to traditional DRAM.In this paper, we propose novel fault-aware WL schemes to allocate write frequencies according to the strength of the rows and handle the imbalance of writes in columns. We use runtime detection schemes to identify weak rows and protect them prior to wear out. In particular, row-level WL, aka RETROFIT, leverages the spare rows provided for redundancy to be used strategically to guard against early cell wear out. RETROFIT is compatible with error correction schemes that guarantee to mitigate hard faults and error-correcting codes (ECC). Rather than discard retired rows, when any spare row completely replaces a retired row, we retarget the retired row to assist with column sparing. It becomes a group of Page Protecting Pointers (PPPs), which utilizes otherwise discarded error correction potential to further enhance the leveling ability of RETROFIT. To relieve column-level imbalance, we apply idle error correction bits before they are used to reduce average bit flips. The evaluation demonstrates that RETROFIT and enhanced RETROFIT with the PPPs improve lifetime by as much as 0.64× and 5.4× in the average case, respectively, over state-of-the-art row-level method while also reducing area overhead. In the worst-case scenario, these improvements further increase to 2.6× and 16.0×. Combined with the proposed column-level WL, enhanced RETROFIT realizes an overall 1.5× memory lifetime improvement over the perfectly uniform wear-leveling with equal storage overhead.

Embedding retrieval is a crucial task for recommendation systems. Graph-based approximate nearest neighbor search (GANNS) is the most commonly used method for retrieval, and achieves the best performance on billion-scale datasets. Unfortunately, the existing CPU- and GPU-based GANNS systems are difficult to optimize the throughput under the latency constraints on billion-scale datasets, due to the underutilized local memory bandwidth (5-45%) and the expensive remote data access overhead (∼ 85% of the total latency). In this paper, we first introduce a practically ideal GANNS architecture for billion-scale datasets, which facilitates a detailed analysis of the challenges and characteristics of distributed GANNS systems. Then, at the architecture level, we propose DF-GAS, a Distributed FPGA-as-a-Service (FPaaS) architecture for accelerating billion-scale Graph-based Approximate nearest neighbor Search. DF-GAS uses a feature-packing memory access engine and a data prefetching and delayed processing scheme to increase local memory bandwidth by 36-42% and reduce remote data access overhead by 76.2%, respectively. At the system level, we exploit the “full-graph + sub-graph” hybrid parallel search scheme on distributed FPaaS system. It achieves million-level query-per-second with sub-millisecond latency on billion-scale GANNS for the first time. Extensive evaluations on million-scale and billion-scale datasets show that DF-GAS achieves an average of 55.4 ×, 32.2 ×, 5.4 ×, and 4.4 × better latency-bounded throughput than CPUs, GPUs, and two state-of-the-art ANNS architectures, i.e., ANNA [23] and Vstore [27], respectively.

Deep Neural Network (DNN) INFerence-as-a-Service (INFaaS) is the dominating workload in current data centers, for which FPGAs become promising hardware platforms because of their high flexibility and energy efficiency. The dynamic and multi-tenancy nature of INFaaS requires careful design in three aspects: multi-tenant architecture, multi-DNN scheduling, and multi-core mapping. These three factors are critical to the system latency and energy efficiency but are also challenging to optimize since they are tightly coupled and correlated. This paper proposesH3M, an automatic Design Space Exploration (DSE) framework to jointly optimize thearchitecture,scheduling, andmappingfor serving INFaaS on cloud FPGAs. H3M explores: (1) the architecture design space withHeterogeneousspatialMulti-tenantsub-accelerators, (2) layer-wise scheduling forHeterogeneousMulti-DNNworkloads, and (3) single-layer mapping to theHomogeneousMulti-corearchitecture. H3M beats state-of-the-art multi-tenant DNN accelerators, Planaria and Herald, by up to 7.5× and 3.6× in Energy-Delay-Product (EDP) reduction on the ASIC platform. On the Xilinx U200 and U280 FPGA platforms, H3M offers 2.1-5.7× and 1.8-9.0× EDP reduction over Herald.

The memristor-based Processing-In-Memory (PIM) architectures have been proven to be a potential architecture to store enormous parameters and execute the complicated computations of Deep Neural Networks (DNNs) efficiently. Existing PIM studies focus on designing high energy-efficient hardware architecture and algorithm-hardware co-optimization for better performance. However, the impacts of the algorithms and hardware architectures on the performance intersect with each other. Only optimizing the algorithms or the hardware architectures can not realize the optimal design. Therefore, the co-exploration of NN models and PIM architecture is necessary. However, for one thing, the co-exploration space size of NN models and PIM architectures is extremely huge, and is challenging to search. For another, during the co-exploration process, time-consuming PIM simulators are needed to evaluate various design candidates and pose a heavy time burden. To tackle these problems, we propose an efficient co-exploration framework of NN models and PIM architectures, named . In, the co-exploration space is carefully designed to adapt both NN models and PIM architectures. Besides, in order to improve search efficiency, we propose an evolutionary search algorithm with adaptive parameter priority (ESAPP). In addition, introduces a multi-level joint simulator to alleviate the problem of time-consuming evaluation. The experimental results show that the proposed co-exploration framework can find better NN models and PIM architectures than existing studies in only six GPU hours (9.8 48.2× speedup). At the same time, can improve the accuracy of co-design results by 15.3% and reduce the energy-delay-product (EDP) by 5.96× compared with existing work.

As a new algorithm of graph embedding, graph neural networks (GNNs) have been widely used in many fields. However, GNN computing has the characteristics of both sparse graph processing and dense neural network, which make it difficult to be deployed efficiently on the existing graph processing accelerators or neural network accelerators. Recently, some GNN accelerators have been proposed, but the following challenges have not been fully solved: 1) the minibatch GNN inference scenario has the potential of software and hardware co-design, which can bring 30% computation amount reduction, and this is not well utilized. Besides, the cost of message flow graph construction is large and may account for more than 50% of the total delay; 2) the feature aggregation has a large amount of data access and relatively small amount of computation, which leads to low on-chip data reuse, only 10% of dense computing; and 3) without the optimization of sparse computing units, simple memory bank and cross bar architecture can easily lead to bank access conflict and load imbalance, reducing the utilization of computing units to less than 60%. In order to solve the above problems, we propose a algorithm-hardware co-design scheme to accelerate GNN inference, which includes three technologies: 1) a reuse-aware sampling method is proposed for minibatch inference scenarios, which reduces 30% of the calculation and improves the on-chip reusability of local data; 2) through the nodewise parallelism-aware quantization, the features and weights are quantized to integers with eight or four bits, which reduces the amount of memory access by at least four times; and 3) an accelerator supporting the above technologies is designed and evaluated, and different operations are supported by the sampling-inference integration architecture. The multibank on-chip memory pool is designed to support data reuse, and edge stream reordering is used to reduce data access conflicts, improving the utilization of computing units by$1.5\times $. Combined with the above technologies, the experiments show that our design achieves$9.2\times $speedup and$29\times $energy efficiency improvement compared with the Deep Graph Library framework running on servers equipped with CPU and GPU.

In the age of Artificial Intelligence (AI), the huge data movements between memory and computing units become the bottleneck of von Neumann architectures, i.e., the “memory wall” problem. In order to tackle this challenge, Processing-In-Memory (PIM) architectures are proposed, which perform in-situ computations in memory and give alternative solutions to boost the computing energy efficiency and performance. Because of the large-scale Neural Network (NN) algorithm models and the huge hardware design space, various factors affect computing accuracy and performance, bringing the need for efficient PIM modeling and evaluation tools. In this work, we propose a behavior-level modeling tool, MNSIM 2.0, to model the performance of PIM architectures efficiently. At the hardware level, MNSIM 2.0 provides a hierarchical PIM modeling structure with flexible architecture configurability and components extensibility. Moreover, the first unified PIM memory array model is proposed for describing both digital and analog PIM. At the algorithm level, MNSIM 2.0 supports the PIM-based NN computing accuracy simulation considering various architecture and device parameters. A PIM-oriented NN model training and quantization flow is also integrated to improve the performance gain brought by PIM. At the scheduling level, MNSIM 2.0 adopts a universal scheduling description compatible with different scheduling strategies. Validation using fabricated PIM macros shows the relative modeling error rate of MNSIM 2.0 is 3:8 5:5%. Case studies show that MNSIM 2.0 enables PIM design space explorations, influences analysis of device parameters, and architecture design insight discoveries.

The memristor-based Processing-In-Memory (PIM) architectures have shown great potential to boost the computing energy efficiency of Neural Networks (NNs). Existing work concentrates on hardware architecture design and algorithm-hardware co-optimization, but neglects the non-negligible impact of the correlation between NN models and PIM architectures. To ensure high accuracy and energy efficiency, it is important to co-design the NN model and PIM architecture. However, on the one hand, the co-exploration space of NN model and PIM architecture is extremely tremendous, making searching for the optimal results difficult. On the other hand, during the co-exploration process, PIM simulators pose a heavy computational burden and runtime overhead for evaluation. To address these problems, in this paper, we propose an efficient co-exploration framework for the NN model and PIM architecture, named Gibbon. In Gibbon, we propose an evolutionary search algorithm with adaptive parameter priority, which focuses on subspace of high priority parameters and alleviates the problem of vast co-design space. Besides, we design a Recurrent Neural Network (RNN) based predictor for accuracy and hardware performances. It substitutes for a large part of the PIM simulator workload and reduces the long simulation time. Experimental results show that the proposed co-exploration framework can find better NN models and PIM architectures than existing studies in only seven GPU hours (8.4~41.3× speedup). At the same time, Gibbon can improve the accuracy of co-design results by 10.7% and reduce the energy-delay-product by 6.48× compared with existing work.

Recently, Graph Convolutional Networks (GCNs) have shown powerful learning capabilities in graph processing tasks. Computing GCNs with conventional von Neumann architectures usually suffers from limited memory bandwidth due to the irregular memory access. Recent work has proposed Processing-In-Memory (PIM) architectures to overcome the bandwidth bottleneck in Convolutional Neural Networks (CNNs) by performing in-situ matrix-vector multiplication. However, the performance improvement and computation parallelism of existing CNN-oriented PIM architectures is hindered when performing GCNs because of the large scale and sparsity of graphs. To tackle these problems, this paper presents a parallelism enhancement framework for PIM-based GCN architectures. At the software level, we propose a fixed-point quantization method for GCNs, which reduces the PIM computation overhead with little accuracy loss. We also introduce the vertex clustering algorithm to the graph, minimizing the inter-cluster links and realizing cluster-level parallel computing on multi-core systems. At the hardware level, we design a Resistive Random Access Memory (RRAM) based multi-core PIM architecture for GCN, which supports the cluster-level parallelism. Besides, we propose a coarse-grained pipeline dataflow to cover the RRAM write costs and improve the GCN computation throughput. At the software/hardware interface level, we propose a PIM-aware GCN mapping strategy to achieve the optimal tradeoff between resource utilization and computation performance. We also propose edge dropping methods to reduce the inter-core communications with little accuracy loss. We evaluate our framework on typical datasets with multiple widely-used GCN models. Experimental results show that the proposed framework achieves$698\times, 89\times$, and$41\times$speedup with$7108\times,255\times$, and$31\times$energy efficiency enhancement compared with CPUs, GPUs, and ASICs, respectively.

Graph mining, which finds specific patterns in the graph, is becoming increasingly important in various domains. We point out that accelerating graph mining suffers from the following challenges: (1) Heavy comparison for pruning: Pruning technique is widely used to reduce search space in graph mining. It applies constraints on vertex indices and involves massive index comparisons. (2) Low parallelism of set operations: The typical graph mining algorithms can be expressed as a series of set operations between neighbors of vertices, which suffer from low parallelism if vertices are streaming to the computation units. (3) Heavy data transfer: Graph mining needs to transfer intermediate data with two orders of magnitude larger than the original data volume between CPU and memory.

Approximate Nearest Neighbor Search (ANNS) is widely used in many fields (e.g., recommender systems). In recent years, the graph-based ANNS methods have attracted the attention of many researchers due to their superiority compared to non-graph-based methods. Compared with traditional recommender systems, mobile recommender systems have higher latency requirements. The graph-based ANNS method faces the following challenges that make it difficult to meet the requirements. (1) Poor connectivity. Due to the limitation of the construction algorithm, the connectivity of the graph is poor, which in turn affects the search performance. (2) Redundant search. The existing search algorithm uses sufficiently long search steps for all queries to achieve high search accuracy. However, the query search steps follow the long-tailed distribution that brings the redundant search, e.g., for more than 40 % of the queries, 87.4 % of the search overhead is redundant. We propose two optimization strategies to tackle the above challenges. (1) Reverse connection enhancement strategy. In the graph construction process, we increase the in-degree of the point to be inserted to enhance the graph connectivity, while keeping the out-degree low to maintain the high search efficiency. (2) Query aware early termination strategy. We identify regional features to predict the number of remaining search steps to achieve dynamic search termination and reduce the redundant search overhead. Finally, we verify the proposed solutions on multiple representative datasets. Compared with the state-of-the-art graph-based algorithm, our solutions can improve the search speed up to 1.21x when the recall rate equals 0.95.

Convolutional neural networks (CNNs) have been widely used in many tasks, but training CNNs is time consuming and energy hungry. Using the low-bit integer format has been proved promising for speeding up and improving the energy efficiency of CNN inference, while CNN training can hardly benefit from such a technique because of the following challenges: 1) the integer data format cannot meet the requirements of the data dynamic range in training, resulting in the accuracy drop; 2) the floating-point data format keeps sizeable dynamic range with much more exponent bits, thus using it results in higher accumulation power than using the integer data format; and 3) there are some specially designed data formats (e.g., with group-wise scaling) that have the potential to deal with the former two problems but common hardware platforms cannot support them efficiently. To tackle all these challenges and make the training phase of CNNs benefit from the low-bit format, we propose a low-bit training framework for CNNs to pursue a better tradeoff between accuracy and energy efficiency: 1) we adopt element-wise scaling to increase the dynamic range of data representation, which significantly reduces the quantization error; 2) group-wise scaling with hardware friendly factor format is designed to reduce the element-wise exponent bits without degrading the accuracy; and 3) we design the customized hardware unit that implements the low-bit tensor convolution arithmetic with our multilevel scaling data format. Experiments show that our framework achieves a superior tradeoff between the accuracy and the bit-width than previous low-bit training studies. For training various models on CIFAR-10, using 1-bit mantissa and 2-bit exponent is adequate to keep the accuracy loss within 1%. On larger datasets like ImageNet, using 4-bit mantissa and 2-bit exponent is adequate. Through the energy consumption simulation of the whole network, we can see that training a variety of models with our framework could achieve$4.9\times $–$10.2\times $higher energy efficiency than full-precision arithmetic.

Memristor based Processing-In-Memory (PIM) systems give alternative solutions to boost the computing energy efficiency of Convolutional Neural Network (CNN) based algorithms. However, Analog-to-Digital Converters' (ADCs) high interface costs and the limited size of the memristor crossbars make it challenging to map CNN models onto PIM systems with both high accuracy and high energy efficiency. Besides, it takes a long time to simulate the performance of large-scale PIM systems, resulting in unacceptable development time for the PIM system. To address these problems, we propose a reliability-aware training framework and a behavior-level modeling tool (MNSIM 2.0) for PIM accelerators. The proposed reliability-aware training framework, containing network splitting/merging analysis and a PIM-based non-uniform activation quantization scheme, can improve the energy efficiency by reducing the ADC resolution requirements in memristor crossbars. Moreover, MNSIM 2.0 provides a general modeling method for PIM architecture design and computation data flow; it can evaluate both accuracy and hardware performance within a short time. Experiments based on MNSIM 2.0 show that the reliability-aware training framework can improve 3.4x energy efficiency of PIM accelerators with little accuracy loss. The equivalent energy efficiency is 9.02 TOPS/W, nearly 2.6~4.2x compared with the existing work. We also evaluate more case studies of MNSIM 2.0, which help us balance the trade-off between accuracy and hardware performance.

Personalized recommendation systems are widely used in many Internet services. The sparse embedding lookup in recommendation models dominates the computational cost of inference due to its intensive irregular memory accesses. Applying resistive random access memory (ReRAM) based process-in-memory (PIM) architecture to accelerate recommendation processing can avoid data movements caused by off-chip memory accesses. However, naïve adoption of ReRAM-based DNN accelerators leads to low computation parallelism and severe under-utilization of computing resources, which is caused by the fine-grained inner-product in feature interaction. In this paper, we propose Rerec, an architecture-algorithm co-designed accelerator, which specializes in fine-grained ReRAM-based inner-product engines with access-aware mapping algorithm for recommendation inference. At the architecture level, we reduce the size and increase the amount of crossbars. The crossbars are fully-connected by Analog-to-Digital Converters (ADCs) in one inner-product engine, which can adapt to the fine-grained and irregular computational patterns and improve the processing parallelism. We further explore trade-offs of (i) crossbar size vs. hardware utilization, and (ii) ADC implementation vs. area/energy efficiency to optimize the design. At the algorithm level, we propose a novel access-aware mapping (AAM) algorithm to optimize resource allocations. Our AAM algorithm tackles the problems of (i) the workload imbalance and (ii) the long recommendation inference latency induced by the great variance of access frequency of embedding vectors. Experimental results show that Rerecachieves 7.69x speedup compared with a ReRAM-based baseline design. Compared to CPU and the state-of-the-art recommendation accelerator, Rerecdemonstrates 29.26x and 3.48x performance improvement, respectively.

With the fast evolvement of embedded deep-learning computing systems, applications powered by deep learning are moving from the cloud to the edge. When deploying neural networks (NNs) onto the devices under complex environments, there are various types of possible faults: soft errors caused by cosmic radiation and radioactive impurities, voltage instability, aging, temperature variations, malicious attackers, and so on. Thus, the safety risk of deploying NNs is now drawing much attention. In this article, after the analysis of the possible faults in various types of NN accelerators, we formalize and implement various fault models from the algorithmic perspective. We propose Fault-Tolerant Neural Architecture Search (FT-NAS) to automatically discover convolutional neural network (CNN) architectures that are reliable to various faults in nowadays devices. Then, we incorporate fault-tolerant training (FTT) in the search process to achieve better results, which is referred to as FTT-NAS. Experiments on CIFAR-10 show that the discovered architectures outperform other manually designed baseline architectures significantly, with comparable or fewer floating-point operations (FLOPs) and parameters. Specifically, with the same fault settings, F-FTT-Net discovered under the feature fault model achieves an accuracy of 86.2% (VS. 68.1% achieved by MobileNet-V2), and W-FTT-Net discovered under the weight fault model achieves an accuracy of 69.6% (VS. 60.8% achieved by ResNet-18). By inspecting the discovered architectures, we find that the operation primitives, the weight quantization range, the capacity of the model, and the connection pattern have influences on the fault resilience capability of NN models.

Convolutional Neural Networks (CNNs) have made breakthroughs in various fields, while the energy consumption becomes enormous. Processing-In-Memory (PIM) architectures based on emerging non-volatile memory (e.g., Resistive Random Access Memory, RRAM) have demonstrated great potential in improving the energy efficiency of CNN computing. However, there is still much room for improvement in the energy efficiency of existing PIM architectures. On the one hand, current work shows that high resolution Analog-to-Digital Converters (ADCs) are required for maintaining computing accuracy, but they dominate more than 60% energy consumption of the entire system, damaging the energy efficiency benefits of PIM. On the other hand, the characteristic of computing in the analog domain in PIM accelerators leads to the computing energy consumption is influenced by the specific input and weight values. However, as far as we know, there is no energy efficiency optimization method based on this characteristic in existing work. To solve these problems, in this paper, we propose an energy-efficient quantized and regularized training framework for PIM accelerators, which consists of a PIM-based non-uniform activation quantization scheme and an energy-aware weight regularization method. The proposed framework can improve the energy efficiency of PIM architectures by reducing the ADC resolution requirements and training low energy consumption CNN models for PIM, with little accuracy loss. The experimental results show that the proposed training framework can reduce the resolution of ADCs by 2 bits and the computing energy consumption in the analog domain by 35%. The energy efficiency, therefore, can be enhanced by $3.4 \times$ in our proposed training framework.

Neural networks (NN) have gained great success in visual object recognition and natural language processing, but this kind of data-intensive applications requires huge data movements between computing units and memory. Emerging resistive random-access memory (RRAM) computing systems have demonstrated great potential in avoiding the huge data movements by performing matrix-vector-multiplications in memory. However, the nonvolatility of the RRAM devices may lead to potential stealing of the NN weights stored in crossbars and the adversary could extract the NN models from the stolen weights. This paper proposes an effective security enhancing method for RRAM computing systems to thwart this sort of piracy attack. We first analyze the theft methods of the NN weights. Then we propose an efficient security enhancing technique based on obfuscating the row connections between positive crossbars and their pairing negative crossbars. Two heuristic techniques are also presented to optimize the hardware overhead of the obfuscation module. Compared with existing NN security work, our method eliminates the additional RRAM writing operations used for encryption/decryption, without shortening the lifetime of RRAM computing systems. The experiment results show that the proposed methods ensure the trial times of brute-force attack are more than (16!)17and the classification accuracy of the incorrectly extracted NN models is less than 20%, with minimal area overhead.

Memristor based neuromorphic computing systems give alternative solutions to boost the computing energy efficiency of Neural Network (NN) algorithms. Because of the large-scale applications and the large architecture design space, many factors will affect the computing accuracy and system's performance. In this work, we propose a behavior-level modeling tool for memristor-based neuromorphic computing systems, MNSIM 2.0, to model the performance and help researchers to realize an early-stage design space exploration. Compared with the former version and other benchmarks, MNSIM 2.0 has the following new features: 1. In the algorithm level, MNSIM 2.0 supports the inference accuracy simulation for mixed-precision NNs considering non-ideal factors. 2. In the architecture level, a hierarchical modeling structure for PIM systems is proposed. Users can customize their designs from the aspects of devices, interfaces, processing units, buffer designs, and interconnections. 3. Two hardware-aware algorithm optimization methods are integrated in MNSIM 2.0 to realize software-hardware co-optimization.

With the in-memory processing ability, ReRAM based computing gets more and more attractive for accelerating neural networks (NNs). However, most ReRAM based accelerators cannot support efficient mapping for sparse NN, and we need to map the whole dense matrix onto ReRAM crossbar array to achieve O(1) computation complexity. In this paper, we propose a sparse NN mapping scheme based on elements clustering to achieve better ReRAM crossbar utilization. Further, we propose crossbar-grained pruning algorithm to remove the crossbars with low utilization. Finally, since most current ReRAM devices cannot achieve high precision, we analyze the effect of quantization precision for sparse NN, and propose to complete high-precision composing in the analog field and design related periphery circuits. In our experiments, we discuss how the system performs with different crossbar sizes to choose the optimized design. Our results show that our mapping scheme for sparse NN with proposed pruning algorithm achieves 3 -- 5X energy efficiency and more than 2.5 -- 6X speedup, compared with those accelerators for dense NN. Also, the accuracy experiments show that our pruning method appears to have almost no accuracy loss.

Convolutional Neural Networks (CNNs) play a vital role in machine learning. Emerging resistive random-access memories (RRAMs) and RRAM-based Processing-In-Memory architectures have demonstrated great potentials in boosting both the performance and energy efficiency of CNNs. However, restricted by the immature process technology, it is hard to implement and fabricate a CNN accelerator chip based on multi-bit RRAM devices. In addition, existing single bit RRAM based CNN accelerators only focus on binary or ternary CNNs which have more than 10% accuracy loss compared with full precision CNNs. This paper proposes a configurable multi-precision CNN computing framework based on single bit RRAM, which consists of an RRAM computing overhead aware network quantization algorithm and a configurable multi-precision CNN computing architecture based on single bit RRAM. The proposed method can achieve equivalent accuracy as full precision CNN but also with lower storage consumption and latency via multiple precision quantization. The designed architecture supports for accelerating the multi-precision CNNs even with various precision among different layers. Experiment results show that the proposed framework can reduce 70% computing area and 75% computing energy on average, with nearly no accuracy loss. And the equivalent energy efficiency is 1.6 ~ 8.6× compared with existing RRAM based architectures with only 1.07% area overhead.

Memristor-based logic design gives an alternative solution to improve the energy efficiency of computing systems, benefiting from combining the memory with computing units. Inspired by this thought, previous work has demonstrated various memristor-based logic families with different attributes and computation patterns. Besides, some logic synthesis tools are designed for specific memristive logic implementations. However, the poor universality and the neglect of realistic constraints in memory largely restrict the utility of these logic synthesis tools. In this paper, we propose a general logic synthesis framework for memristor-based logic design, containing a universal abstract description method for memristive logic, a mapping rules generator, and a synthesis and mapping flow. The proposed logic synthesis framework is suitable for various types of existing memristor-based logic families and takes the memory status into consideration. It is also possible to handle future memristive devices and logic families by providing the universal abstraction interface. Furthermore, we also design a circuit-partitioning-based synthesis acceleration strategy to tackle with the long synthesis time problem. Experimental results show that, our framework can generate mapping results under the restriction of limited resource, while the existing synthesis tools may fail under the same restriction, and achieve comparable synthesis results with the same resource as the existing synthesis tools, which is enough for computation and storage. And the proposed acceleration scheme can achieve ~ 1000× speedup compared with the initial one.

The training of neural networks (NN) is usually time-consuming and resource intensive. The emerging metaloxide resistive random-access memory (RRAM) device has shown potential for the computation of NN. RRAM crossbar structure and multibit characteristics can perform the matrix-vector product in high energy efficiency, which is the most common operation of NN. Two challenges exist for realizing training NN based on RRAM. First, the current architectures based on RRAM only support the inference in training NN and cannot perform the backpropagation (BP) and the weight update of training NN. Second, training NN requires enormous iterations to constantly update the weights for reaching the convergence. However, this weight update leads to large energy consumption because of the nonideal factors of RRAM. In this paper, we propose a training-in-memory based on RRAM (TIME) architecture and the peripheral circuit design to enable training NN on RRAM. TIME supports the BP and the weight update while maximizing the re-usage of peripheral circuits of the inference operation on RRAM. Meanwhile, a set of optimization strategies focusing on the nonideal factors are designed to reduce the cost of tuning RRAM. We explore the performance of both supervised learning (SL) and deep reinforcement learning (DRL) on TIME. A specific mapping method of DRL is also introduced to further improve energy efficiency. Simulation results show that in SL, TIME can achieve 5.3× higher energy efficiency on average compared with DaDianNao, an application-specific integrated circuits (ASIC) in CMOS technology. In DRL, TIME can perform an average 126× higher than GPU in energy efficiency. If the cost of tuning RRAM can be further reduced, TIME has the potential to boost the energy efficiency by two orders of magnitudes compared with ASIC.

Convolutional Neural Networks (CNNs) have achieved excellent performance on various artificial intelligence (AI) applications, while a higher demand on energy efficiency is required for future AI. Resistive Random-Access Memory (RRAM)-based computing system provides a promising solution to energy-efficient neural network training. However, it's difficult to support high-precision CNN in RRAM-based hardware systems. Firstly, multi-bit digital-analog interfaces will take up most energy overhead of the whole system. Secondly, it's difficult to write the RRAM to expected resistance states accurately; only low-precision numbers can be represented. To enable CNN training based on RRAM, we propose a low-bitwidth CNN training method, using low-bitwidth convolution outputs (CO), activations (A), weights (W) and gradients (G) to train CNN models based on RRAM. Furthermore, we design a system to implement the training algorithms. We explore the accuracy under different bitwidth combinations of (A, CO, W, G), and propose a practical tradeoff between accuracy and energy overhead. Our experiments demonstrate that the proposed system perform well on low-bitwidth CNN training tasks. For example, training LeNet-5 with 4-bit convolution outputs, 4-bit weights, 4-bit activations and 4-bit gradients on MNIST can still achieve 97.67% accuracy. Moreover, the proposed system can achieve 23.0X higher energy efficiency than GPU when processing the training task of LeNet-5, and 4.4X higher energy efficiency when processing the training task of ResNet-20.

Emerging memristor devices like metal oxide resistive switching random access memory (RRAM) and memristor crossbar have shown great potential in computing matrix-vector multiplication. However, due to the nonlinear distribution of resistance levels in memristor devices, the state-of-the-art multi-bit cell cannot accomplish the multi-bit computing task accurately. In this paper, we propose fault-tolerant schemes to rescue memristor-based computation with nonlinear resistance levels. We classify the resistance level distributions in memristor devices into three types, and the corresponding models are proposed to analyze the computation characteristics. We propose two theoretical conditions to determine if a memristor device can support multi-bit matrix computation. For the deviated linear model, the least squares method is used to reduce the computing error. When the resistance distribution obeys the proposed power model, a logarithmic operation circuit is used to decode the multiplication results and then accomplish the computing accurately. For the exponential model, since the device cannot complete typical matrix-vector multiplication from hardware level, we propose online and offline quantization methods to make the neural computing algorithms friendly to memristor device. Simulation results show that the root-mean-square error improves around 4% with the linear model and more than 99% with the power model. After quantization, the accuracy of ResNet-18 using memristor with exponential conductance levels can be improved to the same accuracy with ideal linear devices.

Convolutional Neural Networks (CNNs) play a vital role in machine learning. CNNs are typically both computing and memory intensive. Emerging resistive random-access memories (RRAMs) and RRAM crossbars have demonstrated great potentials in boosting the performance and energy efficiency of CNNs. Compared with small crossbars, large crossbars show better energy efficiency with less interface overhead. However, conventional workload mapping methods for small crossbars cannot make full use of the computation ability of large crossbars. In this paper, we propose an Overlapped Mapping Method (OMM) and MIxed Size Crossbar based RRAM CNN Accelerator (MISCA) to solve this problem. MISCA with OMM can reduce the energy consumption caused by the interface circuits, and improve the parallelism of computation by leveraging the idle RRAM cells in crossbars. The simulation results show that MISCA with OMM can achieve 2.7× speedup, 30% utilization rate improvement, and 1.2× energy efficiency improvement on average compared with fixed size crossbars based accelerator using the conventional mapping method. In comparison with GPU platform, MISCA with OMM can perform 490.4× higher on average in energy efficiency and 20× higher on average in speedup. Compared with PRIME, an existing RRAM based accelerator, MISCA has 26.4× speedup and 1.65× energy efficiency improvement.

The training of neural network (NN) is usually time-consuming and resource intensive. Memristor has shown its potential in computation of NN. Especially for the metal-oxide resistive random access memory (RRAM), its crossbar structure and multi-bit characteristic can perform the matrix-vector product in high precision, which is the most common operation of NN. However, there exist two challenges on realizing the training of NN. Firstly, the current architecture can only support the inference phase of training and cannot perform the backpropagation (BP), the weights update of NN. Secondly, the training of NN requires enormous iterations and constantly updates the weights to reach the convergence, which leads to large energy consumption because of lots of write and read operations. In this work, we propose a novel architecture, TIME, and peripheral circuit designs to enable the training of NN in RRAM. TIME supports the BP and the weights update while maximizing the reuse of peripheral circuits for the inference operation on RRAM. Meanwhile, a variability-free tuning scheme and gradually-write circuits are designed to reduce the cost of tuning RRAM. We explore the performance of both SL (supervised learning) and DRL (deep reinforcement learning) in TIME, and a specific mapping method of DRL is also introduced to further improve the energy efficiency. Experimental results show that, in SL, TIME can achieve 5.3x higher energy efficiency on average compared with the most powerful application-specific integrated circuits (ASIC) in the literature. In DRL, TIME can perform averagely 126x higher than GPU in energy efficiency. If the cost of tuning RRAM can be further reduced, TIME have the potential of boosting the energy efficiency by 2 orders of magnitude compared with ASIC.