In recent years, various computing-in-memory (CIM) processors have been presented, showing superior performance over traditional architectures. To unleash the potential of various CIM architectures, such as device precision, crossbar size, and crossbar number, it is necessary to develop compilation tools that are fully aware of the CIM architectural details and implementation diversity. However, due to the lack of architectural support in current popular open-source compiling stacks such as TVM, existing CIM designs either manually deploy networks or build their own compilers, which is time-consuming and labor-intensive. Although some works expose the specific CIM device programming interfaces to compilers, they are often bound to a fixed CIM architecture, lacking the flexibility to support the CIM architectures with different computing granularity. On the other hand, existing compilation works usually consider the scheduling of limited operation types (such as crossbar-bound matrix-vector multiplication). Unlike conventional processors, CIM accelerators are featured by their diverse architecture, circuit, and device, which cannot be simply abstracted by a single level if we seek to fully explore the advantages brought by CIM.

ReRAM-based accelerator of bit-slicing architecture is a promising solution to neural network inference, which allows ineffectual bit-operation removal for greater potential gains. However, existing techniques mostly exploit the removal of weight-associated ineffectual operations, which cannot eliminate the activation-induced ineffectual operations. Alternatively, some approaches adopt an isolated two-stage approach to remove at the weight and activation-level, which leaves a big proportion of ineffectual bit-level operations. Therefore, in contrast to all these coarse-grained operation removal techniques, it is challenging to jointly eliminate ineffectual bit-operation induced by either activation or weight bit-slices for ReRAM-based accelerators. This work presents a novel ineffectual bit-operation removal approach and the accompanied ReRAM-based bit-operation clipping architecture that skips all those bit-level operations that make negligible impacts on neural network outputs. In experiments, the proposed bit-operation clipping ReRAM accelerator, Bit-Trimmer, achieves$5.28 \times$energy efficiency and$2.04 \times$speedup on average. Besides, compared with two SOTA ReRAM accelerator designs with bit-operation removal, it outperforms by$1.56 \times$and$1.88 \times$energy efficiency.

This work proposes a ReRAM-based Computing-in-Memory (CIM) architecture for Neural Architecture Search (NAS) acceleration, ENASA, so that the compute-intensive NAS technology can be applied to various edge devices to customize the most suitable individual solution for their cases. In the popular one-shot NAS process, the system must repetitively evaluate the sampled sub-network within a large-scale supernet before converging to the best sub-network architecture. Thereby, how to map these iterative network inference tasks onto the CIM arrays makes a big difference in system performance. To realize efficient in-memory supernet sampling and evaluation, we design a novel mapping method that tactically executes a group of sub-nets in the CIM arrays, not only to boost the sub-net concurrency but also to eliminate the repetitive operations shared by these subnets. Meanwhile, to further enhance the subnet-level operation concurrency and sharing in the CIM arrays, we also tailor a novel CIM-friendly one-shot NAS algorithm that purposely samples those operation-sharing subnets in each iteration while still maintaining the convergence performance of NAS. According to the experimental results, our CIM NAS accelerator achieves an improvement of 196.6× and 1200× in performance speedup and energy saving respectively compared to the CPU+GPU baseline.

It has recently become trendy for neural network accelerators to scale from single-core to chiplet-based multichip architecture, as the growth of neural network depth and complexity are calling for the promotion of computation and memory capabilities. However, the unintended extensive inter-chip communication of chiplet-based accelerator may bottleneck the parallelism of deep learning inference, which is undesirable for many real-time applications and energy-efficient devices. Although it is imperative for novel schemes to be devised to alleviate this problem, related works are scarce. In this work, we present CampQ, a fine-grained communication-aware mixed-precision quantization method to accelerate inference parallelization by reducing the major inter-chiplet communication overhead. By leveraging the AutoML technique, CampQ is capable of determining different bit-width to activation groups according to thier transmission distances in on-package network. The experimental results show 1.4×-2.6× performance benefits and 29%-60% energy reduction over the 16-bit models for various neural networks and parallelism approaches.

Network sparsity or pruning is a pivotal technology for edge intelligence. Resistive random access memory (RRAM)-based accelerators, featuring dense storage and processing in memory capability, have demonstrated the superior computing performance and energy efficiency over the traditional CMOS-based accelerators for neural network applications. Unfortunately, RRAM-based accelerators suffer the performance or energy degradation when deploying pruned models, impairing their competition in the edge intelligence scenarios. We observed the essential reason is the pruning technology and the mapping strategy in prior RRAM-based accelerator and are optimized individually. As a result, the random zeros in the pruned deep neural network are irregularly distributed in the crossbars, rendering the degradation of computation parallelism of the crossbar without crossbar demand reduction. In this work, we propose a coordinated model pruning and mapping framework to jointly optimize of model accuracy and efficiency of RRAM-based accelerators. As for the mapping, we first decouple weight matrices in the bit-wise way and map the bit matrices to different crossbars, where the signed weights are represented with the two’s complement so as that save half desired crossbars. As for the pruning, we prune weight bits at the crossbar granularity so that free the crossbars holding the pruned bits. Furthermore, we employ an reinforcement learning (RL) approach to automatically select the optimal crossbar-aware bit-pruning strategy for any given neural network without laborious human efforts. We conducted the experiments on a set of representative neural networks and compared our framework with the state-of-the-art (SOTA) bit-sparsity works. The results show that automatic structured bit-pruning saves up to 89.64% energy reduction and 84.12% area overhead compared to existing PRIME-like architecture. Besides, our framework outperforms the SOTA bit-sparsity design by$1.5\times $in terms of the energy reduction on the RRAM-based accelerator.

ReRAM-based accelerators have shown great potential in neural network acceleration via in-memory analog computing. However, high-precision analog-to-digital converters (ADCs), which are required by the ReRAM crossbars to achieve high-accuracy network model inference, play an essential role in the energy-efficiency of the accelerators. Based on the discovery that the ADC precision requirements of crossbars are different, we propose the model-aware crossbarwise ADC precision assignment and the accompanied information-lossless low-bit ADCs to reduce energy overhead without sacrificing model accuracy. In experiments, the proposed information-lossless ReRAM accelerator, InfoX, only consumes 8.97% ADC energy of the SOTA baseline with no accuracy degradation at all.

Real-time ego-motion tracking and 3D structural estimation are the fundamental tasks for the ubiquitous cyper-physical systems, and they can be conducted via the state-of-the-art Edge-Based Visual Odometry (EBVO) algorithm. However, the intrinsic data-intensive process of EBVO emplaces a memory-wall hurdle in practical deployment on conventional von-Neumann-style computing systems. In this work, we attempt to leverage SRAM based processing-in-memory (PIM) technique to alleviate such memory-wall bottleneck, so as to optimize the EBVO systematically from the perspectives of the algorithm layer and physical layer. In the algorithm layer, we first investigate the data reuse patterns of the essential computing kernels required for the feature detection and pose estimation steps in EBVO, and propose PIM friendly data layout and computing scheme for each kernel accordingly. We distill the basic logical and arithmetical operations required in the algorithm layer, and in the physical layer, we propose a novel bit-parallel and reconfigurable SRAM-PIM architecture to realize the operations with high computing precision and throughput. Our experimental result shows that the proposed multi-layer optimization allows for high tracking accuracy of EBVO, and it can improve 11x processing speed and reduce 20x energy consumption compared to the CPU implementation.

Real-time inference of deep learning models on embedded and energy-efficient devices becomes increasingly desirable with the rapid growth of artificial intelligence on edge. Specifically, to achieve superb energy-efficiency and scalability, efficient parallelization of single-pass deep neural network (DNN) inference on chip multiprocessor (CMP) architectures is urgently required by many time-sensitive applications. However, as the number of processing cores scales up and the performance of cores has grown much fast, the on-chip inter-core data movement is prone to be a performance bottleneck for computation. To remedy this problem and further improve the performance of network inference, in this work, we introduce a communication-aware DNN parallelization technique called CAP, by exploiting the elasticity and noise-tolerance of deep learning algorithms on CMP. Moreover, in the hope that the conducted studies can provide new design values for real-time neural network inference on embedded chips, we also have evaluated the proposed approach on both multi-core Neural Network Accelerators (NNA) chips and general-purpose chip-multiprocessors. Our experimental results show that the proposed CAP can achieve 1.12×-1.65× system speedups and 1.14×-2.70× energy efficiency for different neural networks while maintaining the inference accuracy, compared to baseline approaches.

With the advancement of deep neural networks (DNNs), the applications driven by DNNs have been spread from the cloud to the edge. However, the intensive computations and data movements in CNNs impede the adoption of DNNs in resource-constraint edge devices.Quantization, a common model compression method, has attracted a lot of attention as it enables efficient inference by lowering the data bit-width of CNN parameters. Due to the features of massive storage and computing-in-memory array, resistive memory (RRAM) has established the energy efficiency and small area processing-in-memory (PIM) for the acceleration of DNNs at the edge end. However, when deploying the network onto resistive-memory-based PIM (RRAM-based PIM), there will be tremendous unused cells due to the mismatch between the structure of the neural network layer and memory array, resulting in the resource under-utilization and low computation efficiency. In this work, we observed prior quantization approaches fail to improve hardware resource utilization as they ignored the hardware structure information in RRAM. Thus, combining the information of the neural network model and hardware information is essential for a high-utilization RRAM-based PIM design. Considering the vast model parameters and heterogeneous RRAM crossbar structure, we develop a novel quantization framework by leveraging the AutoML technique, i.e., RaQu, which automatically generates a fine-grained quantization strategy for any model that fully utilizes the resource of RRAM-based PIM. The experimental results show that RaQu achieves at most 29.2%–37.4% and 1.8%–3.3% improvement in resource utilization and model accuracy, respectively, compared to prior coarse-grained quantization methods.

Network sparsity or pruning is an extensively studied method to optimize the computation efficiency of deep neural networks (DNNs) for CMOS-based accelerators, such as FPGAs and GPUs. Though the RRAM-based accelerator has demonstrated superior performance and energy efficiency for DNN tasks, deploying the sparse neural networks desires dedicated consideration to save resource consumption without introducing the expensive index overhead and sophisticated control. To exploit the potential of sparse neural network design on the RRAM-based accelerator, we propose an automatic structured bit-pruning design, ASBP, to harmonize the optimization objective of DNN sparsity with efficient RRAM deployment. Specifically, ASBP prunes the bits of weight which are split into different crossbars and thus, free the zero-value crossbar when mapping the neural network into RRAM-based accelerators without extra hardware modification. Meanwhile, ASBP employs the reinforcement learning (RL) approach to automatically select the best crossbar-aware bit-sparsity strategy for any given neural network without laborious human efforts. According to our experiments on a set of representative neural networks, ASBP saves up to 79.01% energy consumption and 54.79% area overhead compared to the baseline that deploys the original DNN on the RRAM-based accelerator. Besides, ASBP outperforms the state-of-the-art bit-sparsity design by 1.4x in terms of the energy reduction on the RRAM-based accelerator.

Convolutional neural networks (CNNs) have become the state-of-the-art technique in many classification tasks in IoT system. However, the low-power and area-constraint edge devices are unable to afford the expensive cost of CNNs. Resistive random access memory (RRAM) is attractive for establishing the CNN accelerator at the edge end due to the features of scalability, low-power and in-situ dot-product. However, mapping a random network architecture onto a general-purpose RRAM accelerator suffers a severe issue of resource underutilization. The neural network quantization offers an opportunity to rescue the degraded resource utilization. Selecting the bit-width for the vast parameters is impractically completed by human labor. This paper proposes an AutoML-based array-aware quantization and mapping framework that generates the fine-grained mixed-precision neural networks to optimize resource utilization in RRAM. In this framework, we design a two-stage learning and array-aware grouping strategy to quickly explore the huge searching space. The experimental results show that the proposed framework achieves 18.2%~36.1% improvement in resource utilization and 0.9%~3.3% increase in model accuracy over prior coarse-grained quantization methods.