Vector Similarity Search (VSS) on edge devices is increasingly essential for data privacy but faces substantial latency and energy overhead due to frequent data transfers between storage and DRAM. To address these challenges, we propose the Intelligent Cognition Engine (ICE), a fully digital non-volatile in-memory computing (nvIMC) framework designed for integration with commercial 3DnandFlash. ICE avoids the use of analog-digital conversion (ADC/DAC) and reduces data movement by performing vector similarity computation directly withinnandstorage. Key architectural components include a digital page multiplier, a two’s-complement accumulator that supports signed computation with minimal circuit modification, and a hierarchical Top-N search strategy to reduce unnecessary data accesses. The proposed framework is evaluated through a combination of post-layout circuit simulations, representative silicon measurements, and system-level modeling on edge platforms. Experimental results indicate that ICE can achieve 17.8–$122.6\times $speedup and 11.5–$162\times $improvement in energy efficiency compared to conventional von Neumann-based approaches, demonstrating the feasibility and scalability of fully digital 3Dnand-based nvIMC for edge AI workloads.

Random forest is a popular ensemble machine-learning algorithm for classification and regression tasks. However, the irregular tree shapes and non-deterministic memory access patterns make it hard for the current von Neumann architecture to handle random forest efficiently. This paper proposes a digital 3D TCAM-based accelerator for the random forest, adopting the idea of processing-in-memory (PIM) to reduce data movement. By utilizing this accelerator, we propose a TCAM-based approach to provide real-time inference with low energy consumption, making it suitable for edge or embedded environments. In the experiments, the proposed approach achieves an average of 3.13 times higher throughput with 22 times more energy saving than the GPU approach.

Vector similarity search (VSS) for unstructured vectors generated via machine learning methods is a promising solution for many applications, such as face search. With increasing awareness and concern about data security requirements, there is a compelling need to store data and process VSS applications locally on edge devices rather than send data to servers for computation. However, the explosive amount of data movement from NAND storage to DRAM across memory hierarchy and data processing of the entire dataset consume enormous energy and require long latency for VSS applications. Specifically, edge devices with insufficient DRAM capacity will trigger data swap and deteriorate the execution performance. To overcome this crucial hurdle, we propose an intelligent cognition engine (ICE) with cognitive 3D NAND, featuring non-volatile in-memory computing (nvIMC) to accelerate the processing, suppress the data movement, and reduce data swap between the processor and storage. This cognitive 3D NAND features digital nvIMC techniques (i. e., ADClDAC-free approach), high-density 3D NAND, and compatibility with standard 3D NAND products with minor modifications. To facilitate parallel INT8/INT4 vector-vector multiplication (VVM) and mitigate the reliability issue of 3D NAND, we develop a bit-error-tolerance data encoding and a two’s complement-based digital accumulator. VVM can support similarity computations (e.g., cosine similarity and Euclidean distance), which are required to search “the most similar data” right where they are stored. In addition, the proposed solution can be realized on edge storage products, e.g., embedded Multi-Media Card (eMMC). The measured and simulated results on real 3D NAND chips show that ICE enhances the system execution time by $17\times to 95\times$ and energy efficiency by $11\times to 140\times$, compared to traditional von Neumann approaches using state-of-the-art edge systems with MobileFaceNet on CASIA-WebFace dataset. To the best of our knowledge, this work demonstrates the first 3D NAND-based digital nvIMC technique with measured silicon data.

Memristor-based deep learning accelerators provide a promising solution to improve the energy efficiency of neuromorphic computing systems. However, the electrical properties and crossbar structure of memristors make these accelerators error-prone. In addition, due to the hardware constraints, the way to deploy neural network models on memristor crossbar arrays affects the computation parallelism and communication overheads. To enable reliable and energy-efficient memristor-based accelerators, a simulation platform is needed to precisely analyze the impact of non-ideal circuit/device properties on the inference accuracy and the influence of different deployment strategies on performance and energy consumption. In this paper, we propose a flexible simulation framework, DL-RSIM, to tackle this challenge. A rich set of reliability impact factors and deployment strategies are explored by DL-RSIM, and it can be incorporated with any deep learning neural networks implemented by TensorFlow. Using several representative convolutional neural networks as case studies, we show that DL-RSIM can guide chip designers to choose a reliability-friendly design option and energy-efficient deployment strategies and develop optimization techniques accordingly.

Computing-In-Memory (CIM) using Flash memory is a potential solution to support a heavy-weight DNN inference accelerator for edge computing applications. Flash memory provides the best high-density and low-cost non-volatile memory solution to store the weights, while CIM functions of Flash memory can compute AI neural network calculations inside the memory chip. Our analysis indicates that Flash CIM can save data movements by ~85% as compared with the conventional Von-Neumann architecture. In this work, we propose a detail device and design co-optimizations to realize Flash CIM, using a novel vertical split-gate Flash device. Our device supports low-voltage (<; 1V) read at WL's and BL's, tight and tunable cell current (Icell) ranging from 150nA to 1.5uA, extremely large Icell ON/OFF ratio ~ 7 orders, small RTN noise and negligible read disturb to provide a high-performance and highly-reliable CIM solution.

Exploiting model sparsity to reduce ineffectual computation is a commonly used approach to achieve energy efficiency for DNN inference accelerators. However, due to the tightly coupled crossbar structure, exploiting sparsity for ReRAM-based NN accelerator is a less explored area. Existing architectural studies on ReRAM-based NN accelerators assume that an entire crossbar array can be activated in a single cycle. However, due to inference accuracy considerations, matrix-vector computation must be conducted in a smaller granularity in practice, called Operation Unit (OU). An OU-based architecture creates a new opportunity to exploit DNN sparsity. In this paper, we propose the first practical Sparse ReRAM Engine that exploits both weight and activation sparsity. Our evaluation shows that the proposed method is effective in eliminating ineffectual computation, and delivers significant performance improvement and energy savings.

Memristor-based deep learning accelerators provide a promising solution to improve the energy efficiency of neuromorphic computing systems. However, the electrical properties and crossbar structure of memristors make these accelerators error-prone. To enable reliable memristor-based accelerators, a simulation platform is needed to precisely analyze the impact of non-ideal circuit and device properties on the inference accuracy. In this paper, we propose a flexible simulation framework, DL-RSIM, to tackle this challenge. DL-RSIM simulates the error rates of every sum-of-products computation in the memristor-based accelerator and injects the errors in the targeted TensorFlow-based neural network model. A rich set of reliability impact factors are explored by DL-RSIM, and it can be incorporated with any deep learning neural network implemented by TensorFlow. Using three representative convolutional neural networks as case studies, we show that DL-RSIM can guide chip designers to choose a reliability-friendly design option and develop reliability optimization techniques.