The emergence of accurate open large language models (LLMs) has sparked a push for advanced quantization techniques to enable efficient deployment on end-user devices. In this paper, we revisit the challenge of extreme LLM compression---targeting ultra-low-bit quantization for both activations and weights---from a Fourier frequency domain perspective. We propose SpecQuant, a two-stage framework that tackles activation outliers and cross-channel variance. In the first stage, activation outliers are smoothed and transferred into the weight matrix to simplify downstream quantization. In the second stage, we apply channel-wise low-frequency Fourier truncation to suppress high-frequency components while preserving essential signal energy, improving quantization robustness. Our method builds on the principle that most of the weight energy is concentrated in low-frequency components, which can be retained with minimal impact on model accuracy. To enable runtime adaptability, we introduce a lightweight truncation module during inference that adjusts truncation thresholds based on channel characteristics. On LLaMA-3 8B, SpecQuant achieves 4-bit quantization for both weights and activations, narrowing the zero-shot accuracy gap to only 1.5% compared to full precision, while delivering 2× faster inference and 3× lower memory usage.

Large Language Models (LLMs) suffer from significant performance and energy efficiency bottlenecks during the memory-bound decoding stage, where GPUs are often underutilized. We propose TFLOP, a novel CPU-FPGA heterogeneous prototype system that addresses this challenge by employing a 4-bit product quantization scheme on model weights and the KV cache. This approach decomposes GEMV operations in decoding stage into two hardware-friendly steps: centroid reconstruction and table lookup, which are efficiently mapped onto an FPGA’s heterogeneous resources. Our key innovation is a unified FPGA architecture that can handle both row- and column-wise quantization, simplifying hardware design and improving efficiency. Evaluations show that TFLOP achieves superior performance, delivering a 2.76$\times$ speedup over the NVIDIA A100 GPU on the LLaMA-2-7B model, while maintaining high accuracy and exceptional energy efficiency.

In recent years, the application of Large Language Models (LLMs) has grown rapidly. LLM inference consists of two stages: the prefill stage and the decoding stage. The prefill stage benefits from high data reuse, allowing GPUs to efficiently utilize computational resources. In contrast, the decoding stage is memory-bound and is more suited for Processing-in-Memory (PIM) techniques. PIM integrates computation units into memory banks to optimize the usage of internal memory bandwidth. However, the limited external bandwidth of PIM creates bottlenecks in two ways. First, PIM systems require high parallelism to fully utilize internal bandwidth, resulting in significant bank-tobank communication. Second, the value cache must be arranged contiguously along the sequence length dimension to maximize DRAM row-buffer hits, which introduces additional transpose overhead. In this work, we propose BLADE, a novel PIM-based architecture designed to accelerate LLM decoding. First, we introduce a task division strategy for multi-head attention (MHA) layers and dynamic PIM parallelism scaling to optimize the balance between computation and communication time. This approach adapts to the increasing sequence length during the decoding process. Second, we leverage the differing DRAM access granularities of CPUs and PIM units to automatically arrange the transposed matrix contiguously in DRAM rows during value cache transfers. Extensive experiments demonstrate that our architecture can significantly reduce the communication overhead and achieve a $105.7 \times$ speedup and $41.6 \times$ energy efficiency compared to the GPU baseline.

Mixture-of-Experts (MoE) models significantly reduce computation in large language models by activating only a subset of experts per input token, but they introduce severe memory bottlenecks due to the large number of expert parameters. Existing offloading and prefetching strategies either incur accuracy loss, prohibitively high memory traffic, or high decoding overhead, limiting deployment on resource-constrained hardware. In this work, we present EARTH, a hardware–software co-design that addresses these challenges through three key innovations. First, we propose a dual-entropy encoding scheme that decomposes each expert into a high-information base and a delta component, enabling compact storage while preserving accuracy via adaptive precision management. Second, we introduce a delta-aware speculative prefetching and reuse mechanism that preloads base components of predicted experts and selectively fetches deltas, reusing previously computed delta patterns to reduce memory traffic and redundant computation. Third, we design a hardware accelerator that is co-designed to efficiently support this encoding and prefetching strategy, optimizing execution order, parallelism, and memory utilization. Across representative MoE workloads, EARTH reduces data movement overhead, improves prefetch efficiency, and achieves up to 2.10× speedup compared to state-of-the-art baselines, while maintaining high model accuracy.

Barrett’s algorithm is one of the most widely used methods for performing modular multiplication, a critical nonlinear operation in modern privacy computing techniques such as homomorphic encryption (HE) and zero-knowledge proofs (ZKP). Since modular multiplication dominates the processing time in these applications, computational complexity and memory limitations significantly impact performance. Computing-in-Memory (CiM) is a promising approach to tackle this problem. However, existing schemes currently suffer from two main problems: 1) Most works focus on low bit-width modular multiplication, which is inadequate for mainstream cryptographic algorithms such as elliptic curve cryptography (ECC) and the RSA algorithm, both of which require high bit-width operations; 2) Recent efforts targeting large number modular multiplication rely on inefficient in-memory logic operations, resulting in high scaling costs for larger bit-widths and increased latency. To address these issues, we propose LaMoS, an efficient SRAM-based CiM design for large-number modular multiplication, offering high scalability and area efficiency. First, we analyze the Barrett’s modular multiplication method and map the workload onto SRAM CiM macros for high bit-width cases. Additionally, we develop an efficient CiM architecture and dataflow to optimize large-number modular multiplication. Finally, we refine the mapping scheme for better scalability in high bit-width scenarios using workload grouping. Experimental results show that LaMoS achieves a 7.02 × speedup and reduces high bit-width scaling costs compared to existing SRAM-based CiM designs.

3D Gaussian Splatting (3DGS) is an emerging neural rendering technique that delivers efficient and high-fidelity rendering, meeting the growing demands of applications such as AR/VR. As 3DGS is increasingly integrated into diverse applications, DNNs are often deployed alongside it to support tasks such as skeletal pose estimation for human avatars or semantic processing for 3D perception. Unfortunately, existing domain-specific accelerators (DSAs) designed for 3DGS excel at rendering but struggle to execute DNN workloads efficiently. Moreover, these DSAs incur significant design and fabrication costs, limiting their practicality. To address these challenges, we propose ORANGE, a novel approach that enables general-purpose DNN-oriented Neural Processing Units (NPUs) to efficiently execute 3DGS without requiring specialized accelerators. The key insight of ORANGE is that we introduce a GEMM-friendly blending process, which reformulates the conventional 3DGS blending operation to fully utilize the matrix multiplication units prevalent in NPUs during rendering. Additionally, to mitigate workload imbalances caused by variable execution latencies across tiles, we develop a sampling-based latency prediction method paired with a tile batching strategy to minimize idle computing resources. Experiments demonstrate that ORANGE achieves up to$1.67 \times$and$15.5 \times$speedup compared to state-of-the-art 3DGS accelerators and the NVIDIA Xavier NX GPU, respectively, in neural rendering tasks. Our approach offers a cost-effective and versatile solution, adhering to the principle of Ockham's Razor by maximizing efficiency without specialized hardware.

Spiking neural networks (SNNs) are an alternative computational paradigm to artificial neural networks (ANNs) that have attracted attention due to their event-driven execution mechanisms, enabling extremely low energy consumption. However, a significant challenge and opportunity in SNNs is to optimize memory and compute costs while maintaining accuracy, thereby further reducing energy consumption. Model quantization has been proposed as a promising technique to improve the running efficiency via the number of data bits reduction. Whereas, this technique has yet to be well studied in the neuromorphic computing domain. The underlying reason is that the behaviors of SNNs are quite different from those of ANNs, making 1) the accuracy of SNNs usually sensitive to data precision, and 2) the introduction of a temporal dimension to characterize neuronal dynamics. In this paper, we present NeuronQuant, an accurate and energy-efficient quantization framework to reduce the precision of neurons while maintaining accuracy. The key insight is to design a post-training quantization method guided by the activity of neurons, efficiently reducing the bit-width of parameters based on local relationships within neurons. Additionally, a budget-aware mixed bit-width allocation strategy for the total model size enables the adaptive growth and narrowing of precision in each layer, leading to a mixed-precision quantization scheme of the desired size. Extensive evaluations demonstrate that NeuronQuant can achieve a compressed SNN with 5.2 bits on average and 1.51× power consumption reduction with a superior model accuracy which is quite impressive for SNN. Code is released at https://github.com/shieldforever/NeuronQuant.

Storing large-scale high-dimensional data, which is rapidly generated by both industry and academia, poses substantial challenges, primarily in terms of storage and maintenance costs. While data compression techniques offer a potential solution to these challenges, they must overcome two critical hurdles: 1) preserving data integrity within lossless bounds and 2) maintaining query performance on compressed data.

Neural Radiance Fields (NeRF) offer significant promise for generating photorealistic images and videos. However, existing mainstream neural rendering models often fall short in meeting the demands for immediacy and power efficiency in practical applications. Specifically, these models frequently exhibit irregular access patterns and substantial computational overhead, leading to undesirable inference latency and high power consumption. Computing-in-memory (CIM), an emerging computational paradigm, has the potential to address these access bottlenecks and reduce the power consumption associated with model execution.

Modular arithmetic, particularly modular reduction, is widely used in cryptographic applications such as homomorphic encryption (HE) and zero-knowledge proofs (ZKP). High-bit-width operations are crucial for enhancing security; however, they are computationally intensive due to the large number of modular operations required. The lookup-table-based (LUT-based) approach, a “space-for-time” technique, reduces computational load by segmenting the input number into smaller bit groups, pre-computing modular reduction results for each segment, and storing these results in LUTs. While effective, this method incurs significant hardware overhead due to extensive LUT usage. In this paper, we introduce ALLMod, a novel approach that improves the area efficiency of LUT-based largenumber modular reduction by employing hybrid workloads. Inspired by the iterative method, ALLMod splits the bit groups into two distinct workloads, achieving lower area costs without compromising throughput. We first develop a template to facilitate workload splitting and ensure balanced distribution. Then, we conduct design space exploration to evaluate the optimal timing for fusing workload results, enabling us to identify the most efficient design under specific constraints. Extensive evaluations show that ALLMod achieves up to $\lt sup\gt1\lt/sup\gt|.65 \times$ and $3 \times$ improvements in area efficiency over conventional LUT-based methods for bit-widths of 128 and 8,192, respectively.

Deep neural networks (DNNs) have revolutionized numerous AI applications, but their vast model sizes and limited hardware resources present significant deployment challenges. Model quantization offers a promising solution to bridge the gap between DNN size and hardware capacity. While INT8 quantization has been widely used, recent research has pushed for even lower precision, such as INT4. However, the presence of outliers-values with unusually large magnitudes-limits the effectiveness of current quantization techniques. Previous compression-based acceleration methods that incorporate outlieraware encoding introduce complex logic. A critical issue we have identified is that serialization and deserialization dominate the encoding/decoding time in these compression workflows, leading to substantial performance penalties during workflow execution. To address this challenge, we introduce a novel computing approach and a compatible architecture design named “BLOOM”. BLOOM leverages the strengths of the “bit-slicing” method, effectively combining structured mixed-precision and bit-level sparsity with adaptive dataflow techniques. The key insight of BLOOM is that outliers require higher precision, while normal values can be processed at lower precision. By interleaving 4-bit values, we efficiently exploit the inherent sparsity in the highprecision components. As a result, the BLOOM-based accelerator outperforms the existing outlier-aware accelerators by an average $1.2 \sim 4.0 \times$ speedup and $24.6 \% \sim 71.3 \%$ energy reduction, respectively, without model accuracy loss.

Large language models (LLMs) are increasingly utilized for complex tasks requiring longer context lengths, with some models supporting up to 128 K or 1 M tokens. This trend, however, presents significant challenges in inference speed and memory management. The primary bottleneck in long-context LLM inference is the quadratic computational complexity of attention mechanisms, causing substantial slowdowns as sequence length increases. KV cache mechanism alleviates this issue by storing pre-computed data, but introduces memory requirements that scale linearly with context length, hindering efficient LLM deployment. Quantization emerges as a promising approach to address the widening gap between LLM size and memory capacity. However, traditional quantization schemes often yield suboptimal compression results for KV caches due to two key factors: i) On-the-fly quantization and de-quantization, causing significant performance overhead; ii) Prevalence of outliers in KV values, challenging low-bitwidth uniform quantization. To this end, we propose MILLION, a novel quantization framework achieving low-bitwidth KV cache through product quantization. First, we conduct a thorough analysis of KV cache distribution, revealing the limitations of existing quantization schemes. Second, we introduce a non-uniform quantization algorithm based on product quantization, which efficiently compresses data while preserving accuracy. Third, we develop a high-performance GPU inference framework with efficient attention kernel and pipeline design for MILLION that leverages sparse computation and asynchronous quantization, significantly enhancing inference speed. Comprehensive evaluation results demonstrate that MILLION can achieve 4 bits quantization with trivial perplexity and accuracy loss, and achieve 2.09 x end-to-end performance gains at 32 K context length. Code is released at https://github.com/ZongwuWang/MILLION.

Large language models (LLMs) have transformed numerous AI applications, with on-device deployment becoming increasingly important for reducing cloud computing costs and protecting user privacy. However, the astronomical model size and limited hardware resources pose significant deployment challenges. Model quantization is a promising approach to mitigate this gap, but the presence of outliers in LLMs reduces its effectiveness. Previous efforts addressed this issue by employing compression-based encoding for mixed-precision quantization. These approaches struggle to balance model accuracy with hardware efficiency due to their value-wise outlier granularity and complex encoding/decoding hardware logic. To address this, we propose PISA (Precision-Slice Framework), an acceleration framework that exploits massive sparsity in the higher-order part of LLMs by splitting 16-bit values into a 4-bit/12-bit format. Crucially, PISA introduces an early bird mechanism that leverages the high-order 4-bit computation to predict the importance of the full calculation result. This mechanism enables efficient computational skips by continuing execution only for important computations and using preset values for less significant ones. This scheme can be efficiently integrated with existing hardware accelerators like systolic arrays without complex encoding/decoding. As a result, PISA outperforms state-of-the-art precision-aware accelerators, achieving a $1.3-4.3 \times$ performance boost and $14.3-66.7 \%$ greater energy efficiency, with minimal model accuracy loss. This approach enables more efficient ondevice LLM deployment, effectively balancing computational efficiency and model accuracy.

Spiking neural networks (SNNs) are an alternative computational paradigm to artificial neural networks (ANNs) that have attracted attention due to their event-driven execution mechanisms, enabling extremely low energy consumption. However, the existing SNN execution model, based on software simulation or synchronized hardware circuitry, is incompatible with the event-driven nature, thus resulting in poor performance and energy efficiency. The challenge arises from the fact that neuron computations across multiple time steps result in increased latency and energy consumption. To overcome this bottleneck and leverage the full potential of SNNs, we propose TAIL, a pioneering temporal asynchronous execution mechanism for SNNs driven by a comprehensive analysis of SNN computations. Additionally, we propose an efficient dataflow design to support SNN inference, enabling concurrent computation of various time steps across multiple layers for optimal Processing Element (PE) utilization. Our evaluations show that TAIL greatly improves the performance of SNN inference, achieving a 6.94× speedup and a 6.97× increase in energy efficiency on current SNN computing platforms.

Hyper-dimensional computing (HDC) is a bio-inspired computing paradigm that mimics cognitive tasks by encoding data into high-dimensional vectors and employing non-complex learning techniques. However, existing HDC solutions face a major challenge hindering their deployment on low-power embedded devices: the costly associative search module, especially in high-precision computations. This module involves calculating the distance between class vectors and query vectors, as well as sorting distances. In this paper, we present HyperDyn, an efficient dynamic inference framework designed for accurate and efficient hyper-dimensional computing. Our framework first offline analyzes the importance of different dimensions in the associative memory based on the contributions of the dimensions to the classification accuracy. In addition, we introduce a dynamic dimensional importance scaling mechanism for more flexible and accurate dimension contribution judgments. Finally, HyperDyn achieves efficient dynamic associative search through a dimension masking mechanism that adapts to the characteristics of the input sample. We evaluate HyperDyn on datasets from three different fields and the results show that HyperDyn can achieve 7.65 × speedup and 58% energy savings, with less than 0.2% loss in accuracy.

Large language models (LLMs) have transformed numerous AI applications, with on-device deployment becoming increasingly important for reducing cloud computing costs and protecting user privacy. However, the astronomical model size and limited hardware resources pose significant deployment challenges. Model quantization is a promising approach to mitigate this gap, but the presence of outliers in LLMs reduces its effectiveness. Previous efforts addressed this issue by employing compression-based encoding for mixed-precision quantization. These approaches struggle to balance model accuracy with hard-ware efficiency due to their value-wise outlier granularity and complex encoding/decoding hardware logic. To address this, we propose OPS (Outlier-aware Precision-Slicing), an acceleration framework that exploits massive sparsity in the higher-order part of LLMs by splitting 16-bit values into a 4-bit/12-bit format. Crucially, OPS introduces an early bird mechanism that leverages the high-order 4-bit computation to predict the importance of the full calculation result. This mechanism enables efficient computational skips by continuing execution only for important computations and using preset values for less significant ones. This scheme can be efficiently integrated with existing hardware accelerators like systolic arrays without complex encoding/decoding. As a result, OPS outperforms state-of-the-art outlier-aware accelerators, achieving a 1.3 − 4.3× performance boost with minimal model accuracy loss. This approach enables more efficient on-device LLM deployment, effectively balancing computational efficiency and model accuracy.

Large language models (LLMs) are increasingly utilized for complex tasks requiring longer context lengths, with some models supporting up to 128K or 1M tokens. This trend, however, presents significant challenges in inference speed and memory management. The primary bottleneck in long-context LLM inference is the quadratic computational complexity of attention mechanisms, causing substantial slowdowns as sequence length increases. KV cache mechanism alleviates this issue by storing pre-computed data, but introduces memory requirements that scale linearly with context length, hindering efficient LLM deployment. Quantization emerges as a promising approach to address the widening gap between LLM size and memory capacity. However, traditional quantization schemes often yield suboptimal compression results for KV caches due to two key factors: i) On-the-fly quantization and de-quantization, causing significant performance overhead; ii) Prevalence of outliers in KV values, challenging low-bitwidth uniform quantization. To this end, we propose EVASION, a novel quantization framework achieving low-bitwidth KV cache through product quantization. First, we conduct a thorough analysis of KV cache distribution, revealing the limitations of existing quantization schemes. Second, we introduce a non-uniform quantization algorithm based on product quantization, which efficiently compresses data while preserving accuracy. Third, we develop a high-performance GPU inference framework for EVASION that leverages sparse computation and asynchronous quantization, significantly enhancing inference speed. Comprehensive evaluation results demonstrate that EVASION can achieve 4 bits quantization trivial perplexity and accuracy loss.

As deep learning models continue to grow larger and more complex, exploiting sparsity is becoming one of the most critical areas for enhancing efficiency and scalability. Several methods for leveraging sparsity have been proposed to more effectively balance the trade-off between compression ratio and accuracy. While these methods offer algorithmic advantages, they also introduce significant hardware overhead due to index-based encoding and decoding. In this paper, we propose CROSS, an end-to-end compilation optimization technique to achieve sparse DNN acceleration using GPU computation kernels. The key insight behind CROSS is to exploit parameter distribution locality and reconcile the “sparse” DNN computation with the high-performance “dense” computation kernels. Specifically, we perform an in-depth analysis of sparse operations in mainstream DNN computing frameworks. We then decompose the sparse workload into multiple components to create highly efficient, specialized operators with different sparsity levels. Additionally, we introduce a novel sparse graph translation technique to facilitate computation kernel processing of the sparse workload. The resulting CROSS framework can accommodate various sparsity patterns and optimization techniques, delivering an average $2.03 \times$ speedup on inference latency compared to seven state-of-the-art solutions with smaller memory footprints across various models and datasets.

DRAM-based processing-in-memory (DRAM-PIM) has gained commercial prominence in recent years. However, its integration for deep learning acceleration, particularly for large language models (LLMs), poses inherent challenges. Existing DRAM-PIM systems are limited in computational capabilities, primarily supporting element-wise and general matrix-vector multiplication (GEMV) operations, which contribute only a small portion of the execution time in LLM workloads. As a result, current systems still require powerful host processors to manage compute-heavy operations.To address these challenges and expand the applicability of commodity DRAM-PIMs in accelerating LLMs, we introduce PLAIN, a novel software/hardware co-design framework for PIM-enabled systems. PLAIN leverages the distribution locality of parameters and the unique characteristics of PIM to achieve optimal trade-offs between inference cost and model quality. Our framework includes three key innovations: 1) firstly, we propose a novel quantization algorithm that determines the optimal precision of parameters within each layer, considering both algorithmic and hardware characteristics to optimize hardware mapping; 2) PLAIN strategically utilizes both GPUs and PIMs, leveraging the high internal memory bandwidth within HBM for attention layers and the powerful compute capability of conventional systems for fully connected (FC) layers; 3) PLAIN integrates a workload-aware dataflow scheduler that efficiently arranges complex computations and memory access for mixed-precision tensors, optimizing execution across different hardware components. Experiments show PLAIN outperforms the conventional GPU with the same memory parameters and the state-of-the-art PIM accelerator, achieving a 5.03× and 1.69× performance boost, with negligible model quality loss.

Transformer-based models have revolutionized computer vision (CV) and natural language processing (NLP) by achieving state-of-the-art performance across a range of benchmarks. However, nonlinear operations in models significantly contribute to inference latency, presenting unique challenges for efficient hardware acceleration. To this end, we propose QUARK, a quantization-enabled FPGA acceleration framework that leverages common patterns in nonlinear operations to enable efficient circuit sharing, thereby reducing hardware resource requirements. QUARK targets all nonlinear operations within Transformer-based models, achieving high-performance approximation through a novel circuit-sharing design tailored to accelerate these operations. Our evaluation demonstrates that QUARK significantly reduces the computational overhead of nonlinear operators in mainstream Transformer architectures, achieving up to a 1.96× end-to-end speedup over GPU implementations. Moreover, QUARK lowers the hardware overhead of nonlinear modules by more than 50% compared to prior approaches, all while maintaining high model accuracy—and even substantially boosting accuracy under ultra-low-bit quantization.

Hyper-Dimensional Computing (HDC) is a promising braininspired learning framework designed for efficient, hardwarefriendly computation.By utilizing highly parallel operations, HDC encodes raw data into a hyper-dimensional space, facilitating efficient training and inference processes.However, the high precision required for representing high-dimensional vectors presents challenges for implementing HDC on resource-constrained edge devices, mainly due to the significant computational cost of performing multiplication for cosine similarity calculations.On the other hand, binary HDC offers lower costs but sacrifices accuracy.This paper addresses these challenges by focusing on data quantization, a hardware-efficient compression technique that incorporates sparsity to effectively balance cost and accuracy on embedded FPGA.Unlike existing methods that employ the same quantization scheme for all dimensions, we propose a novel solution that applies different numerical data types to different dimensions of data representations.This approach is motivated by two factors: * Both authors contributed equally to the paper.

Transformer-based models have demonstrated remarkable performance in computer vision tasks. However, their increasing model size leads to substantial memory demands and higher latency, hindering practical deployment. This paper presents an adaptive dynamic layer-skipping framework based on Markov Decision Process, which determines optimal computational paths based on the current state of input samples. We introduce a Temporal Importance Difference Reward mechanism to address the credit assignment problem in layer-skipping decisions, and develop a knowledge distillation strategy using learnable cognitive tokens to compensate for information loss. Experiments on various models demonstrate that our method significantly reduces computational costs while maintaining accuracy, offering a practical solution for deploying high-performance Transformer models in resource-constrained environments. The code is available at https://github.com/wjjkhl/ASTER

Hyper-Dimensional Computing (HDC) has emerged as a lightweight computational model, renowned for its robust and efficient learning capabilities, particularly suitable for resource-constrained hardware. As HDC often finds its application in edge devices, the associated security challenges pose a critical concern that cannot be ignored. In this work, we aim to quantitatively delve into the robustness of binary HDC, which is widely recognized for its robustness. Employing the bit-flip attack as our initial focal point, we meticulously devise both an attack mechanism and a corresponding defense mechanism. Our objective is to comprehensively explore the robustness of the binary hyper-dimensional computation model, aiming to gain a deeper understanding of its security vulnerabilities and potential defenses. Specifically, we introduce a novel attack framework for HDC, named HyperAttack, which is capable of compromising a robust binary HDC model by maliciously flipping a minimal number of bits within its memory system (specifically, the DRAM) that houses the associative memory. The bit-flip operation is executed through the well-known Row Hammer attack, and HyperAttack optimizes the accuracy degradation by pinpointing the most vulnerable bits in the hyper-dimensional vectors (represented as binary vectors within the associative memory) of the HDC model. The proposed HyperAttack framework is grounded in the principles of fuzziness, seamlessly integrating dimensional ranking and feature similarity analysis within hypervectors to precisely identify the bits to be flipped. Furthermore, we have developed a defense mechanism named HyperDefense, designed to bolster the robustness of binary hyper-dimensional computational models against bit-flip attacks. This defense scheme is tailored specifically for HDC models, providing a robust safeguard against potential threats. HyperDefense operates directly on the associative memory of HDC models, strengthening their defenses. By meticulously modifying selected bits, HyperDefense maintains a high level of accuracy close to the original model, even in the face of increased bit flip rates. This defense mechanism leverages redundant dimensions as backups for critical information. Through a thorough analysis of dimension importance, HyperDefense achieves superior robustness by gracefully sacrificing non-critical dimensions, thus ensuring the model’s robustness against potential attacks.

Sparse matrix-matrix multiplication (SpMM) is a fundamental operation widely used in deep neural networks (DNNs) and high-performance computing. Many compilation studies have optimized the kernel code of SpMM to achieve better performance gains. However, on the one hand, these efforts often focus solely on optimizing individual SpMM operations, without fully considering the influence of preceding and subsequent operators on SpMM. On the other hand, when dense regions in SpMM require accumulation to the same output location, these dense matrix multiplications must be executed sequentially, leading to significant overhead from atomic additions or thread synchronization. In this article, we propose a novel compiler plug-in for efficient SpMM, named SpMMPlu-Pro. SpMMPlu-Pro inherits the sparse intermediate representation (Sparse IR) and sparse pattern representation [meta-operation (meta-op)] as well as five optimization passes from SpMMPlu. To fully utilize the sparse properties, SpMMPlu-Pro implements a forward and backward cross-layer sparsity propagation algorithm, which propagates the sparsity of one layer to the front and back layers, fully releasing the potential of utilizing sparsity to accelerate neural network inference. To alleviate the inefficient accumulation of meta-ops caused by atomic addition or thread synchronization, we propose two complementary scheduling schemes: 1) the segmentation and grouping algorithm based on automatic search and 2) the atomic optimization method through the meta-op data flow graph restructure. We integrated SpMMPlu-Pro into MindSpore and tested its effectiveness and scalability on the NVIDIA V100 GPU and Huawei Ascend 910. The results show that SpMMPlu-Pro supports various sparsity patterns, achieving an average speedup of$4.10\times $on the V100 GPU and$4.35\times $on the Ascend 910 compared to the dense counterpart.

Network sparsification serves as an effective technique to accelerate Deep Neural Network (DNN) inference. However, existing sparsification techniques often rely on structured sparsity, which yields limited benefits. This is primarily due to the significant memory and computational overhead introduced by numerous sparse storage formats during address generation and gradient updates. Additionally, many of these solutions are tailored solely for the inference phase, neglecting the crucial training phase. In this article, we introduce STCO, a novel Sparse Tensor Compilation Optimization technique that significantly enhances training efficiency through structured sparse tensor compilation. Central to STCO is the Tensorization-aware Index Entity (TIE) format, which effectively represents structured sparse tensors by eliminating redundant indices and minimizing storage overhead. The TIE format plays a pivotal role in the Address-carry flow (AC flow) pass, which optimizes the data layout at the computational graph level. This pass leverages the TIE format to enhance the efficiency of tensor representations, enabling more compact and efficient sparse tensor storage. Meanwhile, a shape inference pass utilizes the AC flow to derive optimized tensor shapes, further refining the performance of sparse tensor operations. Moreover, the Address-Carry TIE Flow dynamically tracks nonzero addresses, extending the benefits of sparse optimization to both forward and backward propagation. This seamless integration into the training pipeline enables a smooth transition to sparse tensor compilation without significant modifications to existing codebases. To further boost training performance, we implement an operator-level AC flow optimization pass tailored for structured sparse tensors. This pass generates efficient addresses, ensuring minimal computational overhead during sparse tensor operations. The flexibility of STCO allows it to be efficiently integrated into various frameworks or compilers, providing a robust solution for enhancing training efficiency with structured sparse tensors. Experiments demonstrated that STCO achieved impressive speedups of 3.64×, 5.43×, 4.89×, and 3.91× when compared to state-of-the-art sparse formats on VGG16, ResNet-18, MobileNetV1, and MobileNetV2, respectively. These findings underscore the efficiency and superiority of our proposed approach in leveraging unstructured sparsity for DNN inference acceleration.

Network sparsification is an effective technique for Deep Neural Network (DNN) inference acceleration. However, existing sparsification solutions often rely on structured sparsity, which has limited benefits. This is because many sparse storage formats introduce substantial memory and computation overhead for address generation and gradient update, or they are only applicable during the inference, neglecting the training phase.In this paper, we propose a novel compilation optimization design called TSTC that enables efficient training via structured sparse tensor compilation. TSTC introduces a novel sparse format, Tensorization-aware Index Entity (TIE), that efficiently represents structured sparse tensors by eliminating repeated indices and reducing storage overhead. The TIE format is applied in the Address-carry flow (AC flow) pass, optimizing the data layout at the computational graph layer. Additionally, a shape inference pass utilizes the address-carry flow to derive optimized tensor shapes. Furthermore, an operator-level AC flow optimization pass generates efficient addresses for structured sparse tensors. TSTC is a versatile design that can be efficiently integrated into existing frameworks or compilers. As a result, TSTC achieves 3.64×, 5.43×, 4.89×, and 3.91× speedup compared to state-of-the-art sparse formats on VGG16, ResNet-18, MobileNetV1 and MobileNetV2, respectively.

Hyper-Dimensional Computing (HDC) is a brain-inspired learning framework that is particularly suited to resource-limited edge devices. HDC operates in a high-parallel manner, encoding raw data into hyper-dimensional space, thus enabling efficient training and inference. However, the high dimensionality of data representation in HDC demands a substantial multiplication cost for calculating cosine similarity in high-precision HDC processes. While binarization of HDC can circumvent these multiplications, it often results in unsatisfactory accuracy. In this paper, we propose PAAP-HD, a novel approximation framework that is both accurate and hardware-friendly, designed to enhance the efficiency of HDC inference. Our framework employs a simple neural network as a universal approximator, which can be mapped to parallel Multiply-Accumulate (MAC) operations of the ReRAM-based PIM crossbar. Additionally, we introduce an algorithm to guide model switching, which aids in managing the approximation quality. This algorithm can be instantiated as a just-in-time predictor, seamlessly integrated into HDC to prescribe the appropriate mode for each sample. Our evaluation is conducted on data sets in four different fields, and the results show that PAAP-HD can bring an execution time speedup of 93.1$\times$ and improve energy efficiency by 41.5$\times$ energy with just <1% accuracy loss.

Spiking neural networks (SNNs) are energy-efficient alternatives to commonly used deep artificial neural networks (ANNs). However, their sequential computation pattern over multiple time steps makes processing latency a significant hindrance to deployment. In existing SNNs deployed on time-driven hardware, all layers generate and receive spikes in a synchronized manner, forcing them to share the same time steps. This often leads to considerable time redundancy in the spike sequences and considerable repetitive processing. Motivated by the effectiveness of dynamic neural networks for boosting efficiency, we propose a temporal-wise adaptive SNN, namely TEAS, in which each layer is configured with independent number of time steps to fully exploit the potential of SNNs. Specifically, given an SNN, the number of time steps of each layer is configured according to its contribution to the final performance of the whole network. Then, we exploit the temporal transforming module to produce a dynamic policy that can adapt the temporal information dynamically during inference. The adaptive configuration generating process enables trade-offs between model complexity and accuracy. Through extensive experiments on challenging datasets, we demonstrate that TEAS significantly improves energy efficiency and processing latency while achieving comparable accuracy to state-of-the-art methods.

Deep Neural Network (DNN) inference consumes significant computing resources and development efforts due to the growing model size. Quantization is a promising technique to reduce the computation and memory cost of DNNs. Most existing quantization methods rely on fixed-point integers or floating-point types, which require more bits to maintain model accuracy. In contrast, variable-length quantization, which combines high precision for values with significant magnitudes (i.e., outliers) and low precision for normal values, offers algorithmic advantages but introduces significant hardware overhead due to variable-length encoding and decoding. Also, existing quantization methods are less effective for both (dynamic) activations and (static) weights due to the presence of outliers.

Spiking neural networks (SNNs) are emerging as energy-efficient alternatives to traditional artificial neural networks (ANNs). Their event-driven information processing significantly reduces computational demands while maintaining competitive performance. However, as SNNs are increasingly deployed in edge devices, security concerns have emerged. While significant research efforts have been dedicated to addressing the security vulnerabilities stemming from malicious input, often referred to as adversarial examples, the security of SNN parameters remains relatively unexplored. This work introduces a novel attack methodology for SNNs known as Energy-Oriented SNN attack (EOS). EOS is designed to increase the energy consumption of SNNs through the malicious manipulation of binary bits within their memory systems (i.e., DRAM), where neuronal information is stored. The key insight of EOS lies in the observation that energy consumption in SNN implementations is intricately linked to spiking activity. The bit-flip operation, the well-known Row Hammer technique, is employed in EOS. It achieves this by identifying the most robust neurons in the SNN based on the spiking activity, particularly those related to the firing threshold, which is stored as binary bits in memory. EOS employs a combination of spiking activity analysis and a progressive search strategy to pinpoint the target neurons for bit-flip attacks. The primary objective is to incrementally increase the energy consumption of the SNN while ensuring that accuracy remains intact. With the implementation of EOS, successful attacks on SNNs can lead to an average of 43% energy increase with no drop in accuracy.

Deep Neural Networks (DNNs) have demonstrated remarkable success; however, their increasing model size poses a challenge due to the widening gap between model size and hardware capacity. To address this, model compression techniques have been proposed, but existing compression methods struggle to effectively handle the significant parameter variations (activations and weights) within the model. Moreover, current variance-aware encoding solutions for compression introduce complex logic, leading to limited compression benefits and hardware efficiency. In this context, we present SPARK, a novel algorithm/architecture co-designed solution that utilizes variable-length data representation for local parameter value processing, offering low hardware overhead and high-performance gains. Our key insight is that the high-order part in quantized values are often sparse, allowing us to employ an identity bit to assign the appropriate encoding length, thereby eliminating redundant bit-length footprints. This reduction in data representation based on data characteristics enables a serialized structured data encoding scheme that seamlessly integrates with existing hardware accelerators, such as systolic arrays. We evaluate SPARK-based accelerators against some existing encoding-based accelerator, and our results demonstrate significant improvements. The SPARK-based accelerator achieves up to 4.65 × speedup and 74.7% energy reduction, while maintaining superior model accuracy.

Transformer-based large language models (LLMs) have demonstrated remarkable success; however, their increasing model size poses a challenge due to the widening gap between model size and hardware capacity. To address this, model com-pression techniques have been proposed, but existing compression methods struggle to effectively handle the significant parameter variations (activations and weights) within the model. More-over, current outlier-aware encoding solutions for compression introduce complex logic, leading to limited compression benefits and hardware efficiency. In this context, we present HOLES, a novel algorithm/architecture co-designed solution that utilizes variable-length data representation and metadata for local pa-rameter value processing, offering low hardware overhead and high-performance gains. Our key insight is that only a tiny fraction of activations are outliers demanding high precision representation. This observation enables us to exploit the sparsity of high significant bits within data representation, allowing us to employ an identifier bit to assign the appropriate encoding length, thereby eliminating redundant bit-length footprints. This reduction in data representation based on data characteristics enables a serialized structured data encoding scheme that seam-lessly integrates with existing hardware accelerators, such as systolic arrays. We evaluate HOLES-based accelerators against some existing encoding-based accelerator, and our results demon-strate significant improvements. The HOLES-based accelerator achieves up to 3.98 × speedup and 70.0% energy reduction, while maintaining superior model accuracy.

Spiking neural networks (SNNs) offer computational and energy efficiency advantages over traditional artificial neural networks (ANNs) due to their event-driven representations. Unlike ANNs, which use continuous activation values, SNNs transmit information through spikes-binary events represented as either 0 or 1. This characteristic allows SNNs to replace weight multiplications in ANNs with additions, resulting in a more energy-efficient and less computationally demanding implementation. However, current SNN algorithms prioritize high accuracy and large sparsity by employing complex neuron models with sparse spike generation. This approach tends to compromise energy efficiency and increase latency. On the other hand, existing SNN hardware designs struggle to jointly exploit high parallel processing dataflows and the inherent large sparsity of (dynamic) spikes, due to the unpredictable sparsity patterns and time-dependency of these spikes. To address these issues, this paper proposes PS4, an algorithm-hardware co-design framework. PS4 exploits the inherently rich sparsity in SNN spike activity using a spatial architecture for high energy efficiency and low latency without compromising accuracy. The key insight of PS4 is selectively merging multiple time steps into single-shot computations based on output (latent) sparsity, enabling speculative fast forwarding by skipping iterative spatiotemporal computations across multiple time steps where nothing happens (i.e., output spike is zero). PS4 incorporates lightweight popcount-based circuits to efficiently handle merged time steps, maximizing spike sparsity utilization and hardware parallelism. Thus, the unexplored sparsity of output spikes can be efficiently exploited to achieve highly parallel and energy-efficient computation, with very low overheads. This enables a hardware-efficient PS4 sparsity-exploiting design, which can be efficiently integrated into existing hardware accelerators like systolic arrays. Evaluations show that PS4 outperforms the the state-of-the-art SNN accelerator PTB, PS4 achieves a significant 3.8 × performance gain without compromising energy efficiency. These results showcase the impressive performance and energy efficiency of PS4, making it a compelling choice for SNN inference tasks.

Exploiting sparsity is a key technique to reduce the computation and memory cost attributed to the ever-expanding size of DNN models. Prior sparse DNN accelerators largely exploit structured sparsity, offering limited benefits due to the need to maintain lower sparsity levels to preserve the accuracy of the original models. On the other hand, exploiting unstructured sparsity requires complicated index accesses for non-zeros value. While this approach provides algorithmic advantages, it intro-duces significant hardware overheads due to irregular, largely unpredictable sparsity patterns. As such, it is not hardware-efficient and hence only achieves sub-optimal sparsity-exploiting benefits. To fully unleash the potential of unstructured sparsity, this paper introduces T-BUS, an algorithm and hardware co-design framework for an Efficient Unstructured Sparsity Engine. At the algorithm level, T-BUS proposes a novel sparse encoding format and computation ordering mechanism, reducing computation and storage costs simultaneously. At the hardware level, T-BUS incorporates a specialized parallel lookup structure with a novel dataflow for efficient index-matching operations in bilateral unstructured sparsity computations. Together, these techniques provide a practical approach to harness the highest potential benefits from non-structured sparsity in both storage and computation, while mitigating the challenges associated with unstructured sparsity in hardware design. Compared to existing works, T-BUS achieves up to 85.8% energy saving and 4.72x speedup across workloads with diverse unstructured sparsity levels.

In modern computer systems, the capacity of the translation lookaside buffer (TLB) cannot scale at the same rate as memory capacity. Many workloads, especially those involving large memory, frequently experience TLB misses, making virtual-to-physical address translation a significant performance bottleneck. This issue is even more pronounced on virtualized platforms, such as cloud environments. One major reason for the slow nested or virtualized address translations is that current virtualized systems organize page tables in a multilevel tree structure that is accessed sequentially. As a result, a nested translation may require up to twenty-four sequential memory accesses. To address this challenge, this paper introduces Ninja, a novel hardware-assisted guest page table (gPT) management approach. Ninja leverages hardware to transparently replace the guest physical address (gPA) in frequently accessed gPT entries with the corresponding host physical address (hPA) in the caches. Consequently, Ninja directly offers the guest page table walker with the hPA of the gPT frames, bypassing the traditional gPA⇒hPA translation, thus eliminating the need for nested TLB (NTLB) lookups and significantly reducing the number of nested page table walks. Our design ensures that, from the guest operating system's perspective, the gPT entries still contain gPA, maintaining software transparency. In contrast to software-based shadow paging methods, Ninja eliminates VM-exit overhead and additional DRAM usage. Furthermore, in comparison to other cache-based optimization techniques, Ninja does not incur any additional cache occupancy. Evaluations show that Ninja outperforms the hardware-assisted scheme in modern CPUs by 17%. In comparison to the state-of-the-art Victima design, Ninja also achieves 8.5 % speedup.

DRAM-based Processing in Memory (PIM) addresses the “memory wall” problem by incorporating computing units (PIM units) into main memory devices for faster and wider local data access. However, critical challenges prevent PIM units from being compatible with existing CPU hosts. Memory interleaving and virtual memory limit the size of contiguous data visible to PIM units that constrains the granularity of PIM tasks. Fine-grained PIM tasks result in significant CPU-PIM offloading overhead, offsetting the speed-up of PIM. Existing PIM systems adopt drastic measures to ensure PIM task offloading efficiency, including isolating PIM memory space and turning off global memory interleaving. These interventions, however, decrease the CPU’s memory bandwidth and introduce extra data transfer, leading to an additional “system memory wall”. This new “wall” must be eliminated before fully embracing the PIM technology. In this work, we propose UM-PIM, a PIM system with interleaved CPU pages and non-interleaved PIM pages coexisting in a Uniform and Shared Memory space. UM-PIM enables zero-copy during PIM task offloading and maintains the CPU’s memory bandwidth while ensuring PIM offloading efficiency. Firstly, we propose a dual-track memory management mechanism consisting of independent page allocation and address translation for the two kinds of pages, respectively. Second, we design UM-PIM interface hardware on the DIMM (with PIMs) side to provide a dynamic address mapping for accelerating the data re-layout. Finally, we provide APIs to reduce PIM-to-PIM communication overhead by optimizing the CPU’s access to PIM pages in different communication modes. We compare UM-PIM with a CPU system and the current PIM systems. Results show negligible performance degradation for CPU workloads ($\lt 0.1 \%$) on UM-PIM, contrasting with the $25.8 \%$ degradation on the current PIM system with memory interleaving switched off. For PIM workloads partitioned to CPU and PIM units, UM-PIM can reduce the CPU time by $4.93 \times$, resulting in an end-to-end $1.96 \times$ speedup on average.

Spiking neural networks (SNNs) are considered as energy-efficient alternatives to deep neural networks (DNNs). By adopting event-driven information processing, SNNs can significantly reduce the computational demands associated with DNNs, while still achieving comparable performance. However, current SNNs primarily prioritize high accuracy and large sparsity by constructing complex neuron models that generate sparse spikes. Unfortunately, this approach results in low energy efficiency and high latency, posing a significant challenge for deploying SNNs at the edge. Furthermore, the dominant computation in SNNs, which involves spike-wise Add-Accumulate operations, is well-suited for process-in-memory (PIM) architectures. However, exploiting high parallel processing and spike sparsity in PIM-based SNN accelerators is challenging due to the irregularity and time dependency of spikes.

Brain-inspired spiking neural networks (SNNs) are considered energy-efficient alternatives to conventional deep neural networks (DNNs). By adopting event-driven information processing, SNNs can significantly reduce the computational demands associated with DNNs, while still achieving comparable performance. However, current SNNs primarily prioritize high accuracy by constructing complex neuron models that generate sparse spikes. Unfortunately, this approach results in low energy efficiency and high latency, posing a significant challenge for deploying SNNs at the edge. Furthermore, the dominant computation in SNNs, which involves spike-wise Accumulate-Compare operations, is well-suited for Computing-in-Memory (CIM) architectures. However, exploiting high parallel processing and spike sparsity in CIM-based SNN accelerators is challenging due to the irregularity and time dependency of spikes. To address these limitations, the paper proposes COMPASS, a SRAM-based CIM architecture for efficient SNNs. We first introduce an efficient method to exploit irregular sparsity for both input spikes (explicit) and output spikes (implicit). This is achieved through a speculation mechanism that exploit dynamic spike patterns, enabling lean hardware for sparsity utilization. Additionally, the CIM architecture is carefully modified to facilitate dynamic spike pattern generation and exploitation with minimal overhead. Moreover, we design an adaptive dataflow with temporal spike representation tailored for input/output spikes, reducing memory footprint and enabling parallel execution. Comprehensive evaluation results demonstrate that COMPASS can achieve 26.7x end-to-end speedup over recent SNN accelerators hardware implementation with up to 386.7x less energy per inference.

Resistive Random-Access-Memory (ReRAM) crossbar is one of the most promising neural network accelerators, thanks to its in-memory and in-situ analog computing abilities for Matrix Multiplication-and-Accumulations (MACs). The key limitations are: 1) the number of rows and columns of ReRAM cells for concurrent execution of MACs is constrained, resulting in limited in-memory computing throughput; 2) the cost of high-precision analog-to-digital (A/D) conversions that can offset the efficiency and performance benefits of ReRAM-based Process-In-Memory (PIM). Meanwhile, it is challenging to deploy Deep Neural Network (DNN) models with a large model size in the crossbar since the sparsity of DNNs cannot be effectively exploited in the crossbar structure, especially the sparsity in the activation. As a countermeasure, we develop a novel ReRAM-based PIM accelerator, namely ERA-BS, which pays attention to the correlation between the bit-level sparsity (in both weights and activations) and the performance of the ReRAM-based crossbar. We propose a superior bit-flip scheme combined with the exponent-based quantization, which can adaptively flip the bits of the mapped DNNs to release redundant space without sacrificing the accuracy much or incurring much hardware overhead. Meanwhile, we design an architecture that can integrate the techniques to shrink the crossbar footprint to be used massively. We further propose a dynamic activation sparsity exploitation scheme in conjunction with the tightly coupled structure nature of the crossbar, including crossbar-aware activation pruning and ancillary run-time hardware support. In such a way, we exploit fine-grained sparsity weights (static) and activations (dynamic), respectively, to improve performance while reducing the energy consumption of computation with negligible overheads. Our experiments on a wide variety of networks show that compared to the well-known ReRAM-based PIM accelerator like “ISAAC”, ERA-BS can achieve up to$43\times$,$78\times$, and$73\times$in terms of energy efficiency, area-efficiency, and throughput, respectively. Compared to the state-of-the-art ReRAM-based design “PIM-Prune”, ERA-BS can also achieve$5.3\times$energy efficiency,$7.2\times$area efficiency, and$32\times$performance gain with a similar or even higher accuracy.

Event-driven spiking neural networks (SNNs) have demonstrated significant potential for achieving high energy and area efficiency. However, existing SNN accelerators suffer from issues such as high latency and energy consumption due to serial accumulation-comparison operations. This is mainly because SNN neurons integrate spikes, accumulate membrane potential, and generate output spikes when the potential exceeds a threshold. To address this, one approach is to leverage the sparsity of SNN spikes to reduce the number of time steps. However, this method can result in imbalanced workloads among neurons and limit the utilization of processing elements (PEs). In this paper, we present SATO, a temporal-parallel SNN accelerator that enables parallel accumulation of membrane potential for all time steps. SATO adopts a two-stage pipeline methodology, effectively decoupling neuron computations. This not only maintains accuracy but also unveils opportunities for fine-grained parallelism. By dividing the neuron computation into distinct stages, SATO enables the concurrent execution of spike accumulation for each time step, leveraging the parallel processing capabilities of modern hardware architectures. This not only enhances the overall efficiency of the accelerator but also reduces latency by exploiting parallelism at a granular level. The architecture of SATO includes a novel binary adder-search tree for generating the output spike train, effectively decoupling the chronological dependence in the accumulation-comparison operation. Furthermore, SATO employs a bucket-sort-based method to evenly distribute compressed workloads to all PEs, maximizing data locality of input spike trains. Experimental results on various SNN models demonstrate that SATO outperforms the well-known accelerator, the 8-bit version of “Eyeriss” by$20.7\times$in terms of speedup and$6.0\times$energy-saving, on average. Compared to the state-of-the-art SNN accelerator “SpinalFlow”, SATO can also achieve$4.6\times$performance gain and$3.1\times$energy reduction on average, which is quite impressive for inference.

Bio-plausible spiking neural networks (SNNs) have gained a great momentum due to its inherent efficiency of processing event-driven information. The dominant computation-matrix bit-wise And-Add operations-in SNN is naturally fit for process-in-memory architecture (PIM). The long input spike train of SNN and the bit-serial processing mechanism of PIM, however, incur considerable latency and frequent analog-to-digital conversion, offsetting the performance gain and energy-efficiency. In this paper, we propose a novel Search-in-Memory (SIM) architecture to accelerate the SNN inference, named SIMSnn. Rather than processing the input bit-by-bit over multiple time steps, SIMSnn can take in a sequence of spikes and search the result by parallel associative matches in the CAM crossbar. As a weight-agnostic SNN accelerator, SIMSnn can adapt to various evolving SNNs without rewriting the crossbar array.

Resistive random-access-memory (ReRAM) crossbar is a promising technique for deep neural network (DNN) accelerators, thanks to its in-memory and in-situ analog computing abilities for vector–matrix multiplication-and-accumulations (VMMs). However, it is challenging for crossbar architecture to exploit the sparsity in DNNs. It is inevitably complex and costly to exploit fine-grained sparsity due to the limitation of the tightly coupled crossbar structure. As a countermeasure, we develop a novel ReRAM-based DNN accelerator, named sparse-multiplication-engine (SME), based on a hardware and software co-design framework. First, we orchestrate the bit-sparse pattern to increase the density of bit-sparsity based on existing quantization methods. Such quantized weights can be nicely generated using the alternating direction method of multipliers (ADMM) optimization during the DNN fine-tuning, which can exactly enforce bit patterns in weights. Second, we propose a novel weight mapping mechanism to slice the bits of the weight across crossbars and splice the activation results in peripheral circuits. This mechanism can decouple the tightly coupled crossbar structure and cumulate the sparsity in the crossbar. Finally, a superior squeeze-out scheme empties the crossbars mapped with highly sparse nonzeros from the previous two steps. We design the SME architecture and discuss its use for other quantization methods and different ReRAM cell technologies. We further propose a workload grouping algorithm and a pipeline to achieve workload balance among crossbar-rows that concurrently execute multiply–accumulate operations to optimize the system latency. Putting all together, with the optimized model, compared with prior state-of-the-art designs, the SME shrinks the use of crossbars up to$8.7\times $and$2.1\times $using ResNet-50 and MobileNet-v2, respectively, and achieve average$3.1\times $speed up with no or little accuracy loss on ImageNet.

Spiking Neural Networks (SNNs) have recently attracted enormous research interest since their event-driven and brain-inspired structure enables low-power computation. In image recognition tasks, the best results are achieved by SNN so far utilizing ANN-SNN conversion methods that replace activation functions in artificial neural networks~(ANNs) with integrate-and-fire neurons. Compared to source ANNs, converted SNNs usually suffer from accuracy loss and require a considerable number of time steps to achieve competitive accuracy. We find that the performance degradation of converted SNN stems from the fact that the information capacity of spike trains in transferred networks is smaller than that of activation values in source ANN, resulting in less information being passed during SNN inference. To better correlate ANN and SNN for better performance, we propose a conversion framework to mitigate the gap between the activation value of source ANN and the generated spike train of target SNN. The conversion framework originates from exploring an identical relation in the conversion and exploits temporal separation scheme and novel neuron model for the relation to hold. We demonstrate almost lossless ANN-SNN conversion using SpikeConverter for VGG-16, ResNet-20/34, and MobileNet-v2 SNNs on challenging datasets including CIFAR-10, CIFAR-100, and ImageNet. Our results also show that SpikeConverter achieves the abovementioned accuracy across different network architectures and datasets using 32X - 512X fewer inference time-steps than state-of-the-art ANN-SNN conversion methods.

Binary Neural Networks (BNNs) have attracted tremendous attention in ReRAM-based Process-In-Memory (PIM) systems, since they significantly simplify the hardware-expensive peripheral circuits and memory footprint. Meanwhile, BNNs are proven to have superior bit error tolerance, which inspires us to make use of this capability in PIM systems whose memory bit-cell suffers from severe device defects. Nevertheless, prior works of BNN do not simultaneously meet the criterion that 1) achieving similar accuracy w.r.t its full-precision counterpart; 2) fully binarized without full-precision operation; and 3) rapid BNN construction, which hampers its real-world deployment. This work proposes the first framework called HAWIS, whose generated BNN can satisfy all the above criteria. The proposed framework utilizes the super-net pre-training technique and reinforcement-learning based width search for BNN generation. Our experimental results show that the BNN generated by HAWIS achieves 69.3% top-1 accuracy on ImageNet with ResNet-18. In terms of robustness, our method maximally increases the inference accuracy by 66.9% and 20% compared to 8-bit and baseline 1-bit counterparts under ReRAM non-ideal effects. Our-code is available at: https://github.com/DamonAtSjtu/HAWIS.

Deep hashing has gained growing momentum in large-scale image retrieval. However, deep hashing is computation- and memory-intensive, which demands hardware acceleration. The unique process of hash sequence computation in deep hashing is non-trivial to accelerate due to the lack of an efficient compute primitive for Hamming distance calculation and ranking.

Model compression has been extensively investigated for supporting efficient neural network inference on edge-computing platforms due to the huge model size and computation amount. Recent researches embrace joint-way compression across multiple techniques for extreme compression. However, most joint-way methods adopt a naive solution that applies two approaches sequentially, which can be sub-optimal, as it lacks a systematic approach to incorporate them.

Event-driven spiking neural networks (SNNs) have shown great promise for being strikingly energy-efficient. SNN neurons integrate the spikes, accumulate the membrane potential, and fire output spike when the potential exceeds a threshold. Existing SNN accelerators, however, have to carry out such accumulation-comparison operation in serial. Repetitive spike generation at each time step not only increases latency as well as overall energy budget, but also incurs memory access overhead of fetching membrane potentials, both of which lessen the efficiency of SNN accelerators. Meanwhile, inherent highly sparse spikes of SNNs lead to imbalanced workloads among neurons that hurdle the utilization of processing elements (PEs).

The Resistive Random-Access-Memory (ReRAM) in crossbar structure has shown great potential in accelerating the vector-matrix multiplication, owing to the fascinating computing complexity reduction (from O(n2) to O(1)). Nevertheless, the ReRAM cells still encounter device programming variation and resistance drifting during computation (known as read disturbance), which significantly hamper its analog computing precision. Inspired by prior precise memory programming works, we propose a Self-Terminating Write (STW) circuit for Multi-Level Cell (MLC) ReRAM. In order to minimize the area overhead, the design heavily reuses inherent computing peripherals (e.g., Analog-to-Digital Converter and Trans-Impedance Amplifier) in conventional dot-product engine. Thanks to the fast and precise programming capability of our design, the ReRAM cell can possess 4 linear distributed conductance levels, with minimum latency used for intermediate resistance refreshing. Our comprehensive cross-layer (device/circuit/architecture) simulation indicates that the proposed MLC STW scheme can effectively obtain 2-bit precision via a single programming pulse. Besides, our design outperforms the prior write&verify schemes by 4.7× and 2× in programming latency and energy, respectively.

Models based on the attention mechanism, i.e. transformers, have shown extraordinary performance in Natural Language Processing (NLP) tasks. However, their memory footprint, inference latency, and power consumption are still prohibitive for efficient inference at edge devices, even at data centers. To tackle this issue, we present an algorithm-architecture co-design with dynamic and mixed-precision quantization, DTQAtten. We present empirically that the tolerance to the noise varies from token to token in attention-based NLP models. This finding leads us to quantize different tokens with mixed levels of bits. Thus, we design a compression framework that (i) dynamically quantizes tokens while they are forwarded in the models and (ii) jointly determines the ratio of each precision. Moreover, due to the dynamic mixed-precision tokens caused by our framework, previous matrix-multiplication accelerators (e.g. systolic array) cannot effectively exploit the benefit of the compressed attention computation. We thus design our accelerator with the variable-speed systolic array (VSSA) and propose an effective optimization strategy to alleviate the pipeline-stall problem in VSSA without hardware overhead. We conduct experiments with existing attention-based NLP models, including BERT and GPT-2 on various language tasks. Our results show that DTQAtten outperforms the previous neural network accelerator Eyeriss by 13.12× in terms of speedup and 3.8× in terms of energy-saving. Compared with the state-of-the-art attention accelerator SpAtten, our DTQAtten achieves at least 2.65× speedup and 3.38× energy efficiency improvement.

Current Internet of Things (IoT) embedded applications use machine learning algorithms to process the collected data. However, the computational complexity and storage requirements of existing deep learning methods hinder the wide availability of embedded applications. Spiking Neural Networks (SNN) is a brain-inspired learning methodology that emerged from theoretical neuroscience, as an alternative computing paradigm for enabling low-power computation. Since these IoT devices are usually resource-constrained, compression techniques are crucial in the practical application of SNNs. Most existing methods directly apply pruning methods from artificial neural networks (ANNs) to SNNs, while ignoring the distinction between ANNs and SNNs, thus inhibiting the potential of pruning methods on SNNs. In this paper, inspired by the topology of neuronal co-activity in the neural system, we propose a dynamic pruning framework (dubbed DynSNN) for SNNs, enabling us to seamlessly optimize network topology on the fly almost without accuracy loss. Experimental results on a wide range of classification applications show that the proposed method achieves almost lossless for SNN on MNIST, CIFAR-10, and ImageNet datasets. Moreover, it reaches a ∼0.3% accuracy loss under 34% compression rate on CIFAR and ImageNet, and achieves 60% compression rate with no accuracy loss on MNIST, which reveals remarkable structure refining capability in SNNs.

Spiking Neural Networks (SNNs) have emerged as a promising alternative to traditional deep Artificial Neural Networks (ANNs) due to its power efficiency that stems from their sparse spike-based computation. However, the spike train naturally exhibits high yet unbounded sparsity. This irregularity makes hardware inefficient if deployed directly on existing sparse CNN accelerators that strictly limit the sparsity patterns. Mean-while, SNN inherently contains a large number of redundant connections among neurons, which can be further exploited to reduce the computational burden on model deployment. Therefore, exploiting sparsity is a key technique in accelerating SNN inference on edge devices.To this end, we advocate exploiting irregular sparsity in SNNs for both input spikes (dynamic) and synaptic weights (static) since sparse spikes are inherently distributed in a random pattern and irregular sparsity is more flexible than regular ones. Thus, we propose MISS, a fraMework that takes full advantage of Irregular Sparsity in the SNN through synergistic hardware and software co-design. In the software part, we employ the unstructured pruning on the synaptic weights, eliminating the redundancy in network structure to the greatest extent without affecting the model accuracy. For the hardware part, we also design a sparsity-stationary dataflow that keeps sparse weights stationary in the memory to avoid the decoding overhead. With this dataflow and the matching-based architecture, we can efficiently unify the dynamic and static irregular sparsity to support the neuron computation with a very low overhead. Extensive evaluation on a wide variety of SNNs demonstrates that MISS achieves an average of 36% (up to 57%) improvement in energy efficiency and 23% (up to 48%) speedup over the baseline SNN accelerators.

Weight quantization is well adapted to cope with the ever-growing complexity of the deep neural network (DNN) model. Diversified quantization schemes lead to diverse quantized bit width and formats of the weights, thereby, subject to different hardware implementations. Such variety prevents a general NPU to leverage different quantization schemes to gain performance and energy efficiency. More importantly, a trend of quantization diversity emerges that applies multiple quantization schemes to different fine-grained structures (e.g., a layer or a channel of weight) of a DNN. Therefore, a general architecture is desired to exploit varied quantization schemes. The crossbar-based processing-in-memory (PIM) architecture, a promising DNN accelerator, is well known for its highly efficient matrix-vector multiplication. However, PIM suffers from the inflexible intracrossbar data path because the weight is stationary on the crossbar and binds to the “add” operation along the bitline. Therefore, many nonuniform quantization methods must rollback the quantization before mapping the weights onto the crossbar. Counterintuitively, this article discovers a unique opportunity of the PIM architecture to exploit varied quantization schemes. We first transform the quantization diversity problem into a consistency problem by aligning the bit with the same magnitude along the same bitline of the crossbar. Consequently, such naive weight mapping causes many square hollows of idle PIM cells. We then propose a novel spatial mapping to exempt these “hollow” crossbar from the intercrossbar data path. To further squeeze the weights on fewer crossbars, we decouple the intracrossbar data path from the hardware bitline by a novel temporal scheduling, so that bits with different magnitudes can be placed on cells along the same bitline. Finally, the proposed IVQ includes a temporal pipeline to avoid the introduced stalling cycles, and a data flow with delicate control mechanisms for the new intra and intercrossbar data paths. Putting all together, IVQ achieves$19.7\times $,$10.7\times $,$4.7\times \sim 63.4\times $,$91.7\times $speedup, and$17.7\times $,$5.1\times $,$5.7\times \sim 68.1\times $,$541\times $energy savings over two PIM accelerators (ISAAC and CASCADE), two customized quantization accelerators (based on ASIC and FPGA), and NVIDIA RTX 2080 GPU, respectively.

Most existing RRAM-based designs require expensive analog-to-digital converters (ADCs) digital-to-analog converters (DACs) and excessively occupied crossbars to achieve efficient acceleration. To reduce the overhead of DACs, the existing solution is to split the input into a bit sequence, but the MAC operation that can be completed by one cycle is forced to multiple cycles to the energy-efficiency decrease. For ADCs, it generally partitions the weight into multiple cells, resulting in an excessive number of crossbars or frequent writes on account of insufficient number. To solve this problem, we propose IM3A, an In-Memory Addressing-Assisted Acceleration scheme IM3A decompose MAC operations into multiplication and accumulation, which are implemented separately through the content-addressable and multiply-accumulated capabilities of the crossbar. The energy-efficiency is improved by the CAM crossbar supporting the parallel search of very large numbers of data bits, and the RRAM crossbar selectively enabling the rows to be read based on the hit result of the CAM search. Therefore, only the possibility of operands involved in MAC is deployed on the crossbar. Experimental results show that IM3A applied on various networks achieves system energy-efficiency improvement by 1.7x ∼ 15.9x over two state-of-the-art crossbar accelerators: ISAAC and PIM-Prune.

Resistive Random-Access-Memory (ReRAM) crossbar is one of the most promising neural network accelerators, thanks to its in-memory and in-situ analog computing abilities for Matrix Multiplication-and-Accumulations (MACs). Nevertheless, the number of rows and columns of ReRAM cells for concurrent execution of MACs is constrained, resulting in limited in-memory computing throughput. Moreover, it is challenging to deploy Deep Neural Network(DNN) models with large model size in the crossbar, since the sparsity of DNNs cannot be effectively exploited in the crossbar structure. As the countermeasure, we develop a novel ReRAM-based DNN accelerator, named Bit-Transformer, which pays attention to the correlation between the bit-level sparsity and the performance of the ReRAM-based crossbar. We propose a superior bit-flip scheme combined with the exponent-based quantization, which can adaptively flip the bits of the mapped DNNs to release redundant space without sacrificing the accuracy much or incurring much hardware overhead. Meanwhile, we design an architecture that can integrate the techniques to massively shrink the crossbar footprint to be used. In this way, It efficiently leverages the bit-level sparsity for performance gains while reducing the energy consumption of computation. The comprehensive experiments indicate that our Bit-Transformer outperforms prior state-of-the-art designs up to 13 x, 35 x, and 67 x, in terms of energy-efficiency, area-efficiency, and throughput, respectively. Code will be open-source in the camera-ready version.

Resistive Random-Access-Memory (ReRAM) cross-bar is a promising technique for deep neural network (DNN) accelerators, thanks to its in-memory and in-situ analog computing abilities for Vector-Matrix Multiplication-and-Accumulations (VMMs). However, it is challenging for crossbar architecture to exploit the sparsity in DNNs. It inevitably causes complex and costly control to exploit fine-grained sparsity due to the limitation of tightly-coupled crossbar structure.As the countermeasure, we develop a novel ReRAM-based DNN accelerator, named Sparse-Multiplication-Engine (SME), based on a hardware and software co-design framework. First, we orchestrate the bit-sparse pattern to increase the density of bit-sparsity based on existing quantization methods. Second, we propose a novel weight mapping mechanism to slice the bits of a weight across the crossbars and splice the activation results in peripheral circuits. This mechanism can decouple the tightly-coupled crossbar structure and cumulate the sparsity in the crossbar. Finally, a superior squeeze-out scheme empties the crossbars mapped with highly-sparse non-zeros from the previous two steps. We design the SME architecture and discuss its use for other quantization methods and different ReRAM cell technologies. Compared with prior state-of-the-art designs, the SME shrinks the use of crossbars up to 8.7× and 2.1× using ResNet-50 and MobileNet-v2, respectively, with ≤ 0.3% accuracy drop on ImageNet.

Model quantization has emerged as a mandatory technique for efficient inference with advanced Deep Neural Networks (DNN) by representing model parameters with fewer bits. Nevertheless, prior model quantization either suffers from the inefficient data encoding method thus leading to noncompetitive model compression rate, or requires time-consuming quantization aware training process. In this work, we propose a novel Adaptive Floating-Point (AFP) as a variant of standard IEEE-754 floating-point format, with flexible configuration of exponent and mantissa segments. Leveraging the AFP for model quantization (i.e., encoding the parameter) could significantly enhance the model compression rate without accuracy degradation and model re-training. We also want to highlight that our proposed AFP could effectively eliminate the computationally intensive de-quantization step existing in the dynamic quantization technique adopted by the famous machine learning frameworks (e.g., pytorch, tensorRT, etc.). Moreover, we develop a framework to automatically optimize and choose the adequate AFP configuration for each layer, thus maximizing the compression efficacy. Our experiments indicate that AFP-encoded ResNet-50/MobileNet-v2 only has ∼0.04/0.6% accuracy degradation w.r.t its full-precision counterpart. It outperforms the state-of-the-art works by 1.1% in accuracy using the same bit-width while reducing the energy consumption by 11.2×, which is quite impressive for inference. Code is released at: https://github.com/MXHX7199/ICCV_2021_AFP