Zhirong Chen, Kaiyan Chang, Zhuolin Li, Cangyuan Li, Xinyang He, Chujie Chen, Mengdi Wang, Haobo Xu, Yinhe Han, Huawei Li, Ying Wang. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2026.

Large language model (LLM) inference demands vast memory capacity and hierarchical memory structures, but conventional architectures suffer from excessive energy and latency costs due to frequent data movement across deep memory tiers. To address this, we propose a wafer-scale SRAM-based Computing-in-Memory (CIM) architecture that performs all LLM operations in situ within the first-level SRAM, eliminating off-chip data migration and achieving unprecedented energy efficiency. However, wafer-scale SRAM CIM presents multiple challenges due to the limited first-level memory capacity, which requires efficient compute-memory resource allocation.

Fully Homomorphic Encryption (FHE) is a powerful privacy-preserving technology enabling secure computation on encrypted data, but it suffers from substantial performance overheads. Running FHE efficiently typically requires developing dedicated FHE accelerators, which can be costly and inflexible. Instead of pursuing entirely new accelerators, this paper explores an alternative paradigm: augmenting generic computing devices with a modular FHE-specific hardware extension unit (HEU) to create an efficient FHE engine. To make this paradigm viable, we propose three key innovations: (1) recognizing that some FHE operators are data-intensive and involve a massive volume of ciphertexts, we design the HEU with a 3D stacked memory-based architecture to handle data-intensive operators.We also provide software-level support to facilitate deploying FHE tasks on this extension-based architecture. (2) To capitalize on the hardware parallelism, we propose an adaptive offloading algorithm that intelligently distributes FHE operators between the computing device and the HEU. (3) To optimize the data layout and minimize the inter-tile data communications in the novel 3D stack memory, we propose a dedicated ciphertext mapping mechanism. Experimental results demonstrate that our work achieves substantial acceleration in FHE tasks.

Transformer-based models dominate modern AI workloads but exacerbate memory bottlenecks due to their quadratic attention complexity and ever-growing model sizes. Existing accelerators, such as Groq and Cerebras, mitigate off-chip traffic with large on-chip caches, while algorithmic innovations such as FlashAttention fuse operators to avoid materializing large attention matrices. However, as off-chip traffic decreases, our measurements show that on-chip SRAM accesses account for over 60% of energy in long-sequence workloads, making cache access the new bottleneck. We propose 3D-Flow, a hybrid-bonded, 3D-stacked spatial accelerator that enables register-to-register communication across vertically partitioned PE tiers. Unlike 2D multi-array architectures limited by NoC-based router-to-router transfers, 3D-Flow leverages sub-10 μm vertical TSVs to sustain cycle-level operator pipelining with minimal overhead. On top of this architecture, we design 3D-FlashAttention, a fine-grained scheduling method that balances latency across tiers, forming a bubble-free vertical dataflow without on-chip SRAM round-trips. Evaluations on Transformer workloads (OPT and QWEN models) show that our 3D spatial accelerator reduces 46–93% energy consumption and achieves 1.4×–7.6× speedups compared to state-of-the-art 2D and 3D designs.

Fully Homomorphic Encryption (FHE) is a promising privacy-preserving technique that has drawn increasing attention from academia and industry. It allows computation directly on encrypted data without decryption. However, FHE incurs intensive computations. Chiplet-based designs integrate multiple processors, delivering high performance and thereby are embraced by computation-intensive FHE tasks. Despite the chiplet-based system with various processors, it is designed for unencrypted applications, falling short in handling FHE with unique ciphertext manipulations. One common approach to make it capable of FHE is developing a new FHE accelerator. However, this approach overlooks existing abundant resources already in the system and introduces a large area overhead. In this paper, we propose Chiplever, a framework that empowers a non-FHE-tailored system to efficiently support FHE tasks via a hardware extension. Chiplever aims to leverage the existing resources already in the room for FHE tasks. To achieve this, (1) Chiplever introduces a hardware extension with an FHE unit providing efficient function support for FHE operators. (2) Chiplever proposes an FHE coordinator in the extension, which enables direct ciphertext transfer between the newly introduced extension and existing chiplets, achieving efficient integration of the extension. (3) Chiplever lowers the high-level homomorphic operations to primitive operators that can be matched by existing chiplets and constructs a fine-grained computation graph. Based on this, Chiplever employs a task scheduling algorithm, which partitions the FHE task across the extension and existing chiplets to exploit the parallelism between them and reduce the ciphertext communication overheads. With these hardware and software optimizations, Chiplever achieves efficient FHE acceleration. Compared with prior FHE ASICs, Chiplever achieves 9.6× 15.9× speedup and 6.2× 67.4× throughput improvement on TFHE, while consuming only 18.8% 35.6% of the area overhead of dedicated FHE ASICs.

This is a corrigendum for the article “A data-centric chip design agent framework for Verilog code generation” published in ACM Trans. Des. Autom. Electron Syst. 30, 6, Article 93 (October 2025), 27 pages.

Recent advances of large language models in the field of Verilog generation have raised several ethical and security concerns, such as code copyright protection and dissemination of malicious code. Researchers have employed watermarking techniques to identify codes generated by large language models. However, the existing watermarking works fail to protect RTL code copyright due to the significant syntactic and semantic differences between RTL code and software code in languages such as Python. This paper proposes a hardware watermarking framework RTLMarker that embeds watermarks into RTL code and deeper into the synthesized netlist. We propose a set of rule-based Verilog code transformations, ensuring the watermarked RTL code's syntactic and semantic correctness. In addition, we consider an inherent tradeoff between watermark transparency and watermark effectiveness and jointly optimize them. The results demonstrate RTLMarker's superiority over the baseline in RTL code watermarking.

Quantization is a widely-used compression technology to reduce the overhead of serving large language models (LLMs) on terminal devices and in cloud data centers. However, prevalent quantization methods, such as 8-bit weight-activation or 4-bit weight-only quantization, achieve limited performance improvements due to poor support for low-precision (e.g., 4-bit) activation. This work, for the first time, realizes practical W4A4KV4 serving for LLMs, fully utilizing the INT4 tensor cores on modern GPUs and reducing the memory bottleneck caused by the KV cache. Specifically, we propose a novel fine-grained mixed-precision quantization algorithm (FMPQ) that compresses most activations into 4-bit with negligible accuracy loss. To support mixed-precision matrix multiplication for W4A4 and W4A8, we develop a highly optimized W4Ax kernel. Our approach introduces a novel mixed-precision data layout to facilitate access and fast dequantization for activation and weight tensors, utilizing the GPU's software pipeline to hide the overhead of data loading and conversion. Additionally, we propose fine-grained streaming multiprocessor (SM) scheduling to achieve load balance across different SMs. We integrate the optimized W4Ax kernel into our inference framework, COMET, and provide efficient management to support popular LLMs such as LLaMA-3-70B. Extensive evaluations demonstrate that, when running LLaMA family models on a single A100-80G-SMX4, COMET achieves a kernel-level speedup of 2.88x over cuBLAS and a 2.02x throughput improvement compared to TensorRT-LLM from an end-to-end framework perspective.

Computing-in-memory (CIM) architectures demonstrate superior performance over traditional architectures. To unleash the potential of CIM accelerators, many compilation methods have been proposed, focusing on application scheduling optimization specific to CIM. However, existing compilation methods often overlook CIM's capability to switch dynamically between compute and memory modes, which is crucial for accommodating the diverse memory and computational needs of real-world deep neural network architectures, especially the emerging large language models. To fill this gap, we introduce CMSwitch, a novel compiler to optimize resource allocation for CIM accelerators with adaptive mode-switching capabilities, thereby enhancing the performance of DNN applications. Specifically, our approach integrates the compute-memory mode switch into the CIM compilation optimization space by introducing a new hardware abstraction attribute. Then, we propose a novel compilation optimization pass that identifies the optimal network segment and the corresponding mode resource allocations using dynamic programming and mixed-integer programming. CMSwitch uses the tailored meta-operator to express the compilation result in a generalized manner. Evaluation results demonstrate that CMSwitch achieves an average speedup of 1.31x compared to existing SOTA CIM compilation works, highlighting CMSwitch's effectiveness in fully exploiting the potential of CIM processors for a wide range of real-world DNN applications.

The billion-scale Large Language Models (LLMs) necessitate deployment on expensive server-grade GPUs with large-storage HBMs and abundant computation capability. As LLM-assisted services become popular, achieving cost-effective LLM inference on budget-friendly hardware becomes the current trend. This has sparked extensive research into relocating LLM parameters from expensive GPUs to external host memory. However, the restricted bandwidth between the host and GPU memory limits the inference performance of existing solutions. This work introduces Hermes, a budget-friendly system that leverages the near-data processing units (NDP) within commodity DRAM DIMMs to enhance the performance of a single consumer-grade GPU, achieving efficient LLM inference. We recognize that the inherent activation sparsity in LLMs naturally divides weight parameters into two categories, termed “hot” and “cold” neurons, respectively. Hot neurons, which consist of only approximately 20% of all weight parameters, account for 80% of the total computational load. In contrast, cold neurons make up the other 80% of parameters but are responsible for just 20% of the computational workload. Leveraging this observation, we propose a heterogeneous computing strategy: mapping hot neurons to a single computation-efficient GPU without large-capacity HBMs, while offloading cold neurons to NDP-DIMMs, which offer large memory size but limited computation capabilities. In addition, the dynamic nature of activation sparsity necessitates a real-time partition of hot and cold neurons and adaptive remapping of cold neurons across multiple NDP-DIMM modules. To tackle these issues, we introduce a lightweight predictor that ensures optimal real-time neuron partition and adjustment between GPU and NDP-DIMMs. Furthermore, we utilize a window-based online scheduling mechanism to maintain load balance among multiple NDP-DIMM modules. In summary, Hermes facilitates the deployment of LLaMA2-70B on consumer-grade hardware at a rate of 13.75 tokens/s and realizes an average 75.24 × speedup over the state-of-the-art offloading-based inference system on popular LLMs.

Layout Synthesis for Quantum Computing (LSQC) is a critical component of quantum design tools. Traditional LSQC studies primarily focus on optimizing for reduced circuit depth by adopting a device-centric design methodology. However, these approaches overlook the impact of classical processing and communication time, thereby being insufficient for Dynamic Quantum Circuits (DQC).To address this, we introduce CLASS, a controller-centric layout synthesizer designed to reduce inter-controller communication latency in a distributed control system. It consists of a two-stage framework featuring a hypergraph-based modeling and a heuristic-based graph partitioning algorithm. Evaluations demonstrate that CLASS effectively reduces communication latency by up to 100% with only a 2.10% average increase in the number of additional operations.

One-shot post-training pruning enhances the deployment of billion-scale large language models (LLMs), with the pruning metric playing a pivotal role in determining which weights to remove. However, existing metrics underperform due to their reliance on a simple symbolic combination of weights and activations, overlooking imbalanced weight magnitudes and the disproportionate influence of activation outliers. To overcome these limitations, we introduce \textbf{BaWA}, a novel pruning metric that systematically \underline{Ba}lances \underline{W}eight and \underline{A}ctivation distributions for more effective pruning. BaWA introduces two key innovations: \textbf{magnitude normalization}, which mitigates weight imbalance across channels for fairer pruning decisions, and \textbf{outlier regularization}, which reduces the impact of activation outliers, ensuring more appropriate channel prioritization. To further enhance its effectiveness, BaWA incorporates an efficient and automatic framework for optimizing normalization and regularization hyperparameters. Extensive experiments validate BaWA as a state-of-the-art (SOTA) pruning metric. For instance, applying BaWA to induce 2:4 sparsity in Mistral-7B reduces perplexity in language comprehension by 2.49 and improves average downstream task accuracy by 3.08\%, outperforming the previous SOTA method Wanda.

Due to the demonstrated superiority, dynamic neural networks (NNs), which adapt their network structures to different inputs, have been recognized as an optimized alternative to conventional static NNs. However, researchers have not explored the implications of dynamic NN on neural processing unit (NPU) architecture design. Consequently, we analyze the characteristics and inefficient sources of executing dynamic NNs on existing hardware. From our analysis, existing NPUs, designed for static NNs, cannot effectively handle the execution of dynamic operator and agent-dependent data loading in dynamic NNs.To this end, we present DNA, an efficient accelerator optimized to deal with the challenges of running general dynamic NNs. Firstly, to improve the execution efficiency of dynamic operators, we propose a transverter-based online scheduling strategy to rapidly generate efficient scheduling for each dynamic operator. Secondly, to mitigate hardware idleness caused by the non-deterministic and agent-dependent data access patterns in dynamic NNs, we propose a novel predictor-based prefetching strategy that achieves effective data preloading with negligible cost. We implemented our accelerator, DNA, by integrating an additional online scheduler into a typical many-core baseline accelerator. According to our evaluation of various dynamic NNs, DNA achieves 3.48× speedup and 3.03× energy savings over the baseline accelerator.

Recent advances in large language models (LLMs) have demonstrated significant potential for automated hardware description language (HDL) code generation from high-level specifications. However, two critical challenges limit further progress in this domain: the scarcity of quality Verilog training data and the inability of current approaches to generate RTL code optimized for power, performance, and area (PPA) metrics. This article presents a comprehensive data-centric framework that addresses these limitations through innovations in both pre-fine-tuning data preparation and after-fine-tuning optimization strategies. In the pre-fine-tuning phase, we tackle the data scarcity problem with an automated design-data augmentation framework that generates high-volume, high-quality natural language specifications aligned with corresponding Verilog code and EDA scripts. Our approach creates a complete RTL-level feedback loop by augmenting EDA scripts, RTL code, and EDA tool feedback. In the after-fine-tuning phase, we focus on generating PPA-aware RTL code through a novel search and prompt framework. Our approach implements iterative filtering and selection of LLM-generated Verilog variants while providing high-quality predefined prompts, including composition and interface specifications. To evaluate the effectiveness of our data augmentation method, we fine-tune Llama 2-13B and Llama 2-7B models using the dataset generated by our augmentation framework. The results demonstrate a significant improvement in the Verilog generation tasks with LLMs. Moreover, the accuracy of Verilog generation surpasses that of the current state-of-the-art open-source Verilog generation model, increasing from 58.8% to 70.6% with the same benchmark. Our 13B model has a pass rate improvement compared with GPT-3.5 in Verilog generation and outperforms in EDA script (i.e., SiliconCompiler) generation with only 200 EDA script data. Additionally, to evaluate the effectiveness of the our agent framework, we compare the PPA on the GPT-3.5, where the results show that the agent refined RTL code can have a better quality than the generated RTL code only with GPT-3.5.

Hardware description language (HDL) code designing is a critical component of the chip design process, requiring substantial engineering and time resources. Recent advancements in large language models (LLMs), such as GPT series, have shown promise in automating HDL code generation. However, current LLM-based approaches face significant challenges in meeting real-world hardware design requirements, particularly in handling complex designs and ensuring code correctness. Our evaluations reveal that the functional correctness rate of LLM-generated HDL code significantly decreases as design complexity increases. In this article, we propose the AutoSilicon framework, which aims to scale up the hardware design capability of LLMs. AutoSilicon incorporates an agent system, which (1) allows for the decomposition of large-scale, complex code design tasks into smaller, simpler tasks; (2) provides a compilation and simulation environment that enables LLMs to compile and test each piece of code it generates; and (3) introduces a series of optimization strategies. Experimental results demonstrate that AutoSilicon can scale hardware designs to projects with code equivalent to over 10,000 tokens. In terms of design quality, it further improves the syntax correctness rate and functional correctness rate compared with approaches that do not employ any extensions. For example, compared to directly generating HDL code using GPT-4-turbo, AutoSilicon enhances the syntax correctness rate by an average of 35.8% and improves functional correctness by an average of 35.6%.

The stagnation of Moore’s law stimulates the concept of breaking monolithic chips into smaller chiplets. However, tactic design partitioning remains an unaddressed issue despite its crucial role in chip product cost reduction. In this paper, we propose Chipletizer, a framework to guide the design partitioning for those who would benefit from chiplet reuse across a line of SoC products. The proposed generic framework supports the repartitioning of multiple SoCs into reusable chiplets economically and efficiently with user-specified parameters. Experimental results show that, compared with existing partitioning strategies, our proposed framework achieves notable cost improvement on realistic products with acceptable power and latency overheads.

Recent advances in large language models have demonstrated their potential for automated generation of hardware description language (HDL) code from high-level prompts. Researchers have utilized fine-tuning to enhance the ability of these large language models (LLMs) in the field of Chip Design. However, the lack of Verilog data hinders further improvement in the quality of Verilog generation by LLMs. Additionally, the absence of a Verilog and electronic design automation (EDA) script data augmentation framework significantly increases the time required to prepare the training dataset for LLM trainers. This paper proposes an automated design-data augmentation framework, which generates high-volume and high-quality natural language aligned with Verilog and EDA scripts. For Verilog generation, it translates Verilog files to an abstract syntax tree and then maps nodes to natural language with a predefined template. For Verilog repair, it uses predefined rules to generate the wrong verilog file and then pairs EDA Tool feedback with the right and wrong verilog file. For EDA Script generation, it uses existing LLM(GPT-3.5) to obtain the description of the Script. To evaluate the effectiveness of our data augmentation method, we finetune Llama2--13B and Llama2-7B models using the dataset generated by our augmentation framework. The results demonstrate a significant improvement in the Verilog generation tasks with LLMs. Moreover, the accuracy of Verilog generation surpasses that of the current state-of-the-art open-source Verilog generation model, increasing from 58.8% to 70.6% with the same benchmark. Our 13B model (ChipGPT-FT1) has a pass rate improvement compared with GPT-3.5 in Verilog generation and outperforms in EDA script (i.e., SiliconCompiler) generation with only 200 EDA script data.

Natural language interfaces have exhibited considerable potential in the automation of Verilog generation derived from high-level specifications through the utilization of large language models, garnering significant attention. Nevertheless, this paper elucidates that visual representations contribute essential contextual information critical to design intent for hardware architectures possessing spatial complexity, potentially surpassing the efficacy of natural-language-only inputs. Expanding upon this premise, our paper introduces an open-source benchmark1 for multi-modal generative models tailored for Verilog synthesis from visual-linguistic inputs, addressing both singular and complex modules. Additionally, we introduce an open-source visual and natural language Verilog query language framework to facilitate efficient and user-friendly multi-modal queries. To evaluate the performance of the proposed multi-modal hardware generative AI in Verilog generation tasks, we compare it with a popular method that relies solely on natural language. Our results demonstrate a significant accuracy improvement in the multi-modal generated Verilog compared to queries based solely on natural language. We hope to reveal a new approach to hardware design in the large-hardware-design-model era, thereby fostering a more diversified and productive approach to hardware design.

In this work, we propose Layer-Puzzle, a multi-task allocation and scheduling framework for multi-core NPUs. Based on the proposed latency-prediction model and dynamic parallelization scheme, Layer-Puzzle can generate near-optimal results for each layer under given hardware resources and traffic congestion levels. As an online scheduler, Layer-Puzzle performs a QoS-aware and dynamic scheduling method that picks the superior version from the previously compiled results and co-runs the selected tasks to improve system performance. Our experiments on MLPerf show that Layer-Puzzle can achieve up to 1.61X, 1.53X, and 1.95X improvement in ANTT, STP, and PE utilization, respectively.

Chiplet based multi-die integration has been thought as a key enabler of the agile chip development flow. For 2.5D based multi-die system, Network on Interposer plays an essential role in the performance and the development cost of the chips. This work proposed a reusable NoI design for agile AI chip customization. The proposed NoI design can self-adapt to the inter-die communication patterns of various neural network applications, so the produced interposers can be reused across different AI chip specifications. Experimental results show the proposed NoI design brings 42.7%$\sim$79.5% of total data communication latency reduction in different scenarios, and it also decreased the area overhead by 26.4%.

Multi-task learning systems are commonly adopted in many real-world AI applications such as intelligent robots and self-driving vehicles. Instead of improving single-network performance, this work proposes a specialized Multi-Task Deep Learning Accelerator architecture, MT-DLA, to improve the performance of concurrent networks by exploiting the shared feature and parameters across these models. It is shown in our evaluation with realistic multi-task workloads, MT-DLA dramatically eliminates the memory and computation overhead caused by the shared parameters, activations and computation result. In the experiments with real-world multi-task learning workloads, MT-DLA brings about 1.4x-7.0x energy efficiency boost when compared to the baseline neural network accelerator without multi-task support.

Deep Reinforcement Learning (DRL) is substantially resource-consuming, and it requires large-scale distributed computing-nodes to learn complicated tasks, like videogame and Go play. This work attempts to down-scale a distributed DRL system into a specialized many-core chip and achieve energy-efficient on-chip DRL. With the customized Network-on-Chip that handles the communication of on-chip data and control-signals, we proposed a Synchronous Asynchronous RL Architecture (SARLA) and the according many-core chip that completely avoids the unnecessary data duplication and synchronization activities in multi-node RL systems. In evaluation, the SARLA system achieves considerable energy-efficiency boost over the GPU-based implementations for typical DRL workloads built with OpenAI-gym.