Multichiplet deep neural network (DNN) accelerators have evolved as promising modular solutions, offering enhanced performance, scalability, and cost-effectiveness. These architectures, however, suffer from the escalating communication bottleneck with increasing scale, primarily stemming from rising hop count and worsening link underutilization. The bottleneck is exacerbated by conventional fixed interconnection networks’ nonadaptability to diverse DNN dataflows. Moreover, an efficient routing tailored for large-scale networks with deadlock-freedom is needed for performance. To address these scalability challenges, leveraging the low-latency links and abundant interconnection resources with reconfigurability in the active interposer, we propose HiRe, a hierarchical reconfigurable network-routing co-design architecture for large-scale multichiplet DNN accelerators. The architecture introduces reconfigurable nodes (RNs) across on-chip and interchiplet hierarchical networks, enabling dynamic bypassing and network reconfiguration. Based on the network, it incorporates an efficient deadlock-free routing that combines simulated annealing (SA)-based communication scheduling with greedy path selection. Through the network-routing co-design, HiRe reduces the hop count and enhances the link utilization. The HiRe architecture is implemented and synthesized in a 55-nm CMOS process. Experimental results demonstrate that HiRe achieves a 14.3%–45.2% EDP reduction and a 14.1%–35.0% latency reduction compared to state-of-the-art (SOTA) innovations, effectively mitigating the large-scale communication bottleneck.

Large Language Models (LLMs) have demonstrated unprecedented generative performance across a wide range of applications. While recent heterogeneous architectures attempt to address the memory-bound bottleneck from attention computations by processing-in-memory (PIM) offloading, they overlook two critical characteristics of attention GEMVs that distinguish them from traditional PIM scenarios: (1) dynamic matrix dimensions that scale with token length, and (2) distinct GEMV patterns between score computation ($Q \times K_{t}$) and context computation ($S \times V$). Existing PIM designs, employing either uniform or transposed computing modes, suffer from inefficiencies in newly generated element preparation or distinct GEMV execution. To address these limitations, we propose AttenPIM, a software-hardware co-design for efficient PIM-based attention acceleration. For bank-level execution, we propose dual-mode computing modes tailored for score and context computations with PIM-oriented data layouts and execution flows for KV storage, supported by a low-cost configurable per-bank PIM unit (PU). For system-level execution, we leverage token-level and head-level concurrency to ensure workload balance and maximize bank PU parallelism. Furthermore, dynamic allocation and kernel fusion methods are proposed to further minimize memory overhead. Experimental results demonstrate that AttenPIM achieves $1.13 \times-5.26 \times$ speedup and reduces energy consumption by 17 %-49 % compared to two state-of-the-art PIM baselines.

With the widespread deployment of long-context large language models (LLMs), efficient and high-quality generation is becoming increasingly important. Modern LLMs employ batching and key-value (KV) cache to improve generation throughput and quality. However, as the context length and batch size rise drastically, the KV cache incurs extreme external memory access (EMA) issues. Recent LLM accelerators face substantial processing element (PE) under-utilization due to the low arithmetic intensity of attention with KV cache, while existing KV cache compression algorithms struggle with hardware inefficiency or significant accuracy degradation. To address these issues, an algorithm-architecture co-optimization, KVO-LLM, is proposed for long-context batched LLM generation. At the algorithm level, we propose a KV cache quantization-aware pruning method that first adopts salient-token-aware quantization and then prunes KV channels and tokens by attention guided pruning based on salient tokens identified during quantization. Achieving substantial savings on hardware overhead, our algorithm reduces the EMA of KV cache over 91% with significant accuracy advantages compared to previous KV cache compression algorithms. At the architecture level, we propose a multi-core jointly optimized accelerator that adopts operator fusion and cross-batch interleaving strategy, maximizing PE and DRAM bandwidth utilization. Compared to the state-of-the-art LLM accelerators, KVO-LLM improves generation throughput by up to $7.32 \times$, and attains $5.52 \sim 8.38 \times$ better energy efficiency.

Bit-serial computation shows promise for accelerating deep neural networks (DNNs) by exploiting inherent bit sparsity. However, the original unstructured bit sparsity poses two major challenges for existing bit-serial accelerators (BSA): (1) workload imbalance from irregular bit distribution, and (2) inefficient memory access due to unpredictable non-zero bit locations. To address these issues, this paper proposes BitPattern, an algorithm/hardware co-design to efficiently accelerate bitserial computation through bit-pattern pruning. At the algorithm level, we employ bit-pattern pruning to identify optimal combinations of predefined patterns and apply compression encoding to minimize weight storage. We further devise a pattern-similaritybased merging method to balance the bit-serial workload. At the hardware level, we co-design a bit-serial accelerator with a dedicated bit-pattern decoder and PE to leverage the potential of structured bit-pattern sparsity. The evaluation on several deep learning benchmarks shows that BitPattern can achieve $1.72 \times$ memory reduction with negligible accuracy loss, and up to $2.11 \times$ speedup and $1.86 \times$ energy saving compared to state-of-the-art bit-serial accelerators.

Spiking neural networks (SNNs) are an alternative computational paradigm to artificial neural networks (ANNs) that have attracted attention due to their event-driven execution mechanisms, enabling extremely low energy consumption. However, the existing SNN execution model, based on software simulation or synchronized hardware circuitry, is incompatible with the event-driven nature, thus resulting in poor performance and energy efficiency. The challenge arises from the fact that neuron computations across multiple time steps result in increased latency and energy consumption. To overcome this bottleneck and leverage the full potential of SNNs, we propose TAIL, a pioneering temporal asynchronous execution mechanism for SNNs driven by a comprehensive analysis of SNN computations. Additionally, we propose an efficient dataflow design to support SNN inference, enabling concurrent computation of various time steps across multiple layers for optimal Processing Element (PE) utilization. Our evaluations show that TAIL greatly improves the performance of SNN inference, achieving a 6.94× speedup and a 6.97× increase in energy efficiency on current SNN computing platforms.

Hyper-dimensional computing (HDC) is a bio-inspired computing paradigm that mimics cognitive tasks by encoding data into high-dimensional vectors and employing non-complex learning techniques. However, existing HDC solutions face a major challenge hindering their deployment on low-power embedded devices: the costly associative search module, especially in high-precision computations. This module involves calculating the distance between class vectors and query vectors, as well as sorting distances. In this paper, we present HyperDyn, an efficient dynamic inference framework designed for accurate and efficient hyper-dimensional computing. Our framework first offline analyzes the importance of different dimensions in the associative memory based on the contributions of the dimensions to the classification accuracy. In addition, we introduce a dynamic dimensional importance scaling mechanism for more flexible and accurate dimension contribution judgments. Finally, HyperDyn achieves efficient dynamic associative search through a dimension masking mechanism that adapts to the characteristics of the input sample. We evaluate HyperDyn on datasets from three different fields and the results show that HyperDyn can achieve 7.65 × speedup and 58% energy savings, with less than 0.2% loss in accuracy.

DIMM-based near-memory processing (NMP) architectures address the “memory wall” problem by incorporating near-memory accelerators (NMAs) into main memory devices for high memory bandwidth and low energy consumption. However, critical challenges prevent efficient asynchronous execution between host and NMAs in DIMM-NMP architectures. Memory controllers (MCs) distributed at the host side and the NMA side issue memory accesses independently without synchronization on memory states, which may lead to memory bus contention and DRAM errors. Therefore, most existing DIMM-NMP designs adopt synchronous execution to prevent concurrent memory accesses. However, this intervention wastes either the host or the NMA computation capability. In this work, we propose AsyncDIMM, a novel DIMM-NMP design with efficient asynchronous execution based on existing memory buses. It enables single access mode (host or NMA), concurrent access mode, and a seamless switch between them. First, we propose the offload-schedule-return mechanism with explicit and implicit synchronization to ensure memory access correctness for all memory modes. Second, to further improve bandwidth utilization and decrease access latency, we introduce optimized timing constraints for offloading, a locality-aware switch-recovery method for scheduling, and adaptive batch with timing-division multiplexing notification for returning. Finally, we present a detailed design with limited hardware modifications to conventional host and NMA MCs, which is extensively validated on the FPGA. Comprehensive experiments demonstrate that AsyncDIMM outperforms four NMP baselines by $1.19 \times-1.92 \times$, enabling efficient asynchronous execution with up to $2.25 \times$ bandwidth utilization uplift and 47% access latency reduction.

Hyper-Dimensional Computing (HDC) is a promising braininspired learning framework designed for efficient, hardwarefriendly computation.By utilizing highly parallel operations, HDC encodes raw data into a hyper-dimensional space, facilitating efficient training and inference processes.However, the high precision required for representing high-dimensional vectors presents challenges for implementing HDC on resource-constrained edge devices, mainly due to the significant computational cost of performing multiplication for cosine similarity calculations.On the other hand, binary HDC offers lower costs but sacrifices accuracy.This paper addresses these challenges by focusing on data quantization, a hardware-efficient compression technique that incorporates sparsity to effectively balance cost and accuracy on embedded FPGA.Unlike existing methods that employ the same quantization scheme for all dimensions, we propose a novel solution that applies different numerical data types to different dimensions of data representations.This approach is motivated by two factors: * Both authors contributed equally to the paper.

Near data processing (NDP), enabled by near data accelerators (NDAs) within DIMM-based main memory, enhances performance by providing more aggregated bandwidth and reducing long-distance data transfers. While the performance of NDAs has received widespread attention, the overhead of host-NDA communication has been overlooked, becoming a bottleneck in NDP systems. To alleviate performance degradation from communication, we propose Bridge-NDP, the first NDP architecture that implements a workflow with efficient communication-computation overlap. Bridge-NDP is built upon the conventional NDP architecture and can be easily applied to existing NDP designs, regardless of the memory level where NDAs are attached. Specifically, we introduce a novel direct host-NDA communication method that utilizes existing memory buses as bridge buses, avoiding the need for new interconnections. It enables seamless integration with other memory accesses while achieving high bandwidth utilization with minimal hardware overhead. For the system-level workflow design, we optimize and extend existing dataflow to achieve richer computing paradigms with fewer redundant memory accesses. Additionally, we provide programming support with efficient API designs and data management to hide low-level resource details and ensure correctness guarantees. Comprehensive experiments demonstrate that Bridge-NDP achieves significant performance improvements, with speedups of$1.8\times $–$3.1\times $and bandwidth utilization improvement of$2.0\times $–$2.9\times $over the state-of-the-art NDP solutions.

Neural radiance field has become a fundamental rendering technique across diverse applications such as augmented/virtual reality and autonomous driving. It achieves exceptional rendering quality and reduces model construction cost mainly by introducing a novel neural representation, Instant Neural Graphics Primitives (Instant-NGP). Despite its superiority, Instant-NGP poses severe problems of intensive computation, memory inefficiency and pipeline inefficiency, owing to numerous neural network queries, irregular memory access and intricate sampling procedure. To address these issues, this article proposes NeRA, an algorithm-architecture co-optimization framework that facilitates the efficient neural rendering of Instant-NGP. For intensive computation, we reconstruct the rendering flow and propose a deferred neural decoding algorithm to aggregate the network queries, which reduces the computational workload by 85.6% and only incurs <0.5 % loss in rendering quality. For memory inefficiency, a base-interpolation algorithm is proposed to condense the scattered memory access and improves the equivalent bandwidth of on-chip memory by 2.38×. Furthermore, a voxel-centric data flow is proposed to fully reuse the cached data and save 88.7% of the external memory access. For pipeline inefficiency, a highly-pipelined hardware architecture with decoupled spatial skipping and interleaved sampling is constructed to eliminate the bubbles and invalid samples in the pipeline, which boosts the overall throughput by 2.41×. Extensively evaluated on representative benchmarks, NeRA attains 1.2 2.9× in rendering throughput, 1.7 36.5× in energy-efficiency and 3.6 8.3× in area-efficiency, compared to the state-of-the-art related architectures.

Vision-centric 3D perception has become a key mechanism in autonomous driving. It achieves exceptional perceptual performance mainly by introducing a novel attention,multi-view cross-attention(MVCA), for learnable feature extraction and fusion from surround-view cameras. Despite its superiority, MVCA encounters severe inefficiencies in sample, processing elements (PE), and pipelined processing, owing to the redundant and non-uniform sampling-aggregation and rigorous inter-operator dependencies. To address these issues, this article proposes a dedicated MVCA accelerator, MVAtor, with algorithm-architecture co-optimization for vision-centric 3D perception based on multi-view inputs flexibly. For sample inefficiency, a 3-tier hybrid static-dynamic sample and a sensitivity-aware feature pruning approach are proposed to eliminate the 86.03% sample overhead and 24.48% memory requirement, only incuring <1% accuracy loss with no need of fine-tuning. For PE inefficiency, a spatial pruner and sequential sampler collaboration strategy is proposed to improve the sampler utilization without compromising pruner’s throughput, which outperforms the previous design by 53.7~96.1% energy-delay product reduction. For pipeline inefficiency, a fine-grained-tiling assisted highly-pipelined architecture is constructed in MVAtor by exploiting the decoupling opportunities on inter-view sparsity, thereby saving 61.03% external memory access while boosting the overall throughputs by 1.83×. Extensively evaluated on representative benchmarks, MVAtor attains 1.38~7.67× and 1.67~11.15× improvement on energy and area efficiency respectively, compared to the state-of-the-art related accelerators.

Large Language Models (LLMs) have achieved significant success in various Natural Language Processing (NLP) tasks, becoming essential to modern intelligent computing. Their large memory footprint and high computational cost hinder efficient deployment. Post-Training Quantization (PTQ) is a promising technique to alleviate this issue and accelerate LLM inference. However, the presence of outliers impedes the advancement of LLM quantization to lower bit levels. In this paper, we introduce OFQ-LLM, an algorithm-hardware co-design solution that adopts outlier-flexing quantization to efficiently accelerate LLM at low-bit levels. The key insight of OFQ-LLM is that normal data can be efficiently quantized in a slightly reduced data encoding space, while the rest encoding space can be used for flexible outlier values. During quantization, we use rescale-based clipping (RBC) to optimize accuracy for normal data and group outlier clustering (GOC) to flexibly represent outlier values. At the hardware level, we introduce a memory-aligned outlier-flexing encoding scheme to encode activations and weights in LLMs at a low bit level. The outlier-normal mixed hardware architecture is devised to leverage the encoding scheme and accelerate LLMs with high speed and high energy efficiency. Our experiments show that OFQ-LLM achieves better accuracy compared to state-of-the-art (SOTA) low-bit LLM PTQ works. OFQ-LLM-based accelerator surpasses the SOTA outlier-aware accelerators by up to$2.69\times $core energy efficiency, up to$3.83\times $speed up and$2.44\times $energy reduction in LLM prefilling phase, and up to$2.01\times $speed up and$2.88\times $energy reduction in LLM decoding phase, with superior accuracy.

Multi-scale deformable attention (MSDeformAttn) has emerged as a key mechanism in various vision tasks, demonstrating explicit superiority attributed to multi-scale grid-sampling. However, this newly introduced operator incurs irregular data access and enormous memory requirement, leading to severe PE under-utilization. Meanwhile, existing approaches for attention acceleration cannot be directly applied to MSDeformAttn due to lack of support for this distinct procedure. Therefore, we propose a dedicated algorithm-architecture co-design dubbed DEFA, the first-of-its-kind method for MSDeformAttn acceleration. At the algorithm level, DEFA adopts frequency-weighted pruning and probability-aware pruning for feature maps and sampling points respectively, alleviating the memory footprint by over 80%. At the architecture level, it explores the multi-scale parallelism to boost the throughput significantly and further reduces the memory access via fine-grained layer fusion and feature map reusing. Extensively evaluated on representative benchmarks, DEFA achieves 10.1-31.9X speedup and 20.3-37.7X energy efficiency boost compared to powerful GPU platforms. It also rivals the related accelerators by 2.2-3.7X energy efficiency improvement while providing pioneering support of MSDeformAttn.

Vision Transformers (ViTs) have achieved remarkable success in computer vision (CV) and are increasingly recognized as the new backbone for vision-language multi-modal tasks. Despite their success, the high computational cost associated with ViTs hinders their inference efficiency. In this paper, we introduce BSViT, a bit-serial Vision Transformer accelerator enhanced by algorithm-hardware co-design. BSViT can efficiently accelerate both plain and hierarchical Vision Transformer inference. At the algorithm level, we propose a post-training quantization scheme named dynamic patch and weight bit-group quantization. We first introduce a dynamic patch quantization (DPQ) scheme to dynamically allocate bit-width to different image patches based on their importance, thus reducing bit width and saving computation without significantly impacting accuracy. Second, we propose a weight bit-group quantization (BGQ) scheme to evenly distribute bits within groups and achieve workload balance across processing elements (PEs). At the hardware level, we propose a term-separate bit-serial accelerator to efficiently support DPQ and BGQ. We introduce dense and sparse bit-serial PEs to manipulate the dense least significant term (LST) and sparse most significant term (MST) workloads. A dense-sparse hybrid dataflow is devised to efficiently balance the two kinds of workloads. Our experiments show that BSViT can achieve up to$1.95\times $speedup and$2.72\times $energy efficiency compared to state-of-the-art (SOTA) bit-serial accelerators and achieve up to$3.69\times $energy efficiency compared to SOTA Transformer accelerators.

With the advancement of artificial intelligence, the collaboration of multiple deep neural networks (DNNs) has been crucial to existing embedded systems and cloud systems, especially for automatic driving applications as well as augmented and virtual reality (AR/VR) applications. To trade off between cost and performance, chiplet-based DNN accelerators have emerged as a promising solution for accelerating DNN workloads. However, most existing mapping methods for multiple DNNs target for the monolithic chip, which fail to solve the problems faced by the emerging multi-chiplet architecture, such as the problems of distributed memory access, complex heterogeneous interconnect network, and the scaling-up of computing resources. In this work, we propose M2M, a fine-grained mapping framework for accelerating multiple DNNs on a multi-chiplet architecture. It includes a temporal and spatial task scheduling for reconfigurable dataflow accelerators and a communication-aware task mapping in a heterogeneous interconnect network. To enhance communication efficiency and reduce the overall latency, we further propose a fine-tuned quality-of-service (QoS) policy for network-on-package (NoP) links. To the best of our knowledge, this is the first fine-grained mapping framework for multiple DNNs on a multi-chiplet architecture. We implemented the proposed fine-grained mapping framework using genetic algorithm and simulated annealing algorithm. Experimental results show that our work achieves 7.18%–61.09% latency reduction under vision, language, and mixed workloads when compared with the state-of-the-art related work.

Grid-based feature learning network plays a key role in recent point-cloud based 3D perception. However, high point sparsity and special operators lead to large memory footprint and long processing latency, posing great challenges to hardware acceleration. We propose FLNA, a novel feature learning accelerator with algorithm-architecture co-design. At algorithm level, the dataflow-decoupled graph is adopted to reduce 86% computation by exploiting inherent sparsity and concat redundancy. At hardware design level, we customize a pipelined architecture with block-wise processing, and introduce transposed SRAM strategy to save 82.1% access power. Implemented on a 40nm technology, FLNA achieves 13.4 − 43.3× speedup over RTX 2080Ti GPU. It rivals the state-of-the-art accelerator by 1.21× energy-efficiency improvement with 50.8% latency reduction.

Point-cloud-based 3D perception has attracted great attention in various applications including robotics, autonomous driving and AR/VR. In particular, the 3D sparse convolution (SpConv) network has emerged as one of the most popular backbones due to its excellent performance. However, it poses severe challenges to real-time perception on general-purpose platforms, such as lengthy map search latency, high computation cost, and enormous memory footprint. In this paper, we propose SpOctA, a SpConv accelerator that enables high-speed and energy-efficient point cloud processing. SpOctA parallelizes the map search by utilizing algorithm-architecture co-optimization based on octree encoding, thereby achieving 8.8-21.2× search speedup. It also attenuates the heavy computational workload by exploiting inherent sparsity of each voxel, which eliminates computation redundancy and saves 44.4-79.1% processing latency. To optimize on-chip memory management, a SpConv-oriented non-uniform caching strategy is introduced to reduce external memory access energy by 57.6% on average. Implemented on a 40nm technology and extensively evaluated on representative benchmarks, SpOctA rivals the state-of-the-art SpConv accelerators by 1.1-6.9× speedup with 1.5-3.1× energy efficiency improvement,