Transformer-based Large Language Models rely critically on the KV cache to efficiently handle extended contexts during the decode phase. Yet, the size of the KV cache grows proportionally with the input length, burdening both memory bandwidth and capacity as decoding progresses. To address this challenge, we present RocketKV, a training-free KV cache compression strategy containing two consecutive stages. In the first stage, it performs coarse-grain permanent KV cache eviction on the input sequence tokens. In the second stage, it adopts a hybrid sparse attention method to conduct fine-grain top-k sparse attention, approximating the attention scores by leveraging both head and sequence dimensionality reductions. We show that RocketKV provides a compression ratio of up to 400×, end-to-end speedup of up to 3.7× as well as peak memory reduction of up to 32.6% in the decode phase on an NVIDIA A100 GPU compared to the full KV cache baseline, while achieving negligible accuracy loss on a variety of long-context tasks. We also propose a variant of RocketKV for multi-turn scenarios, which consistently outperforms other existing methods and achieves accuracy nearly on par with an oracle top-k attention scheme. The source code is available here: https://github.com/NVlabs/RocketKV.

In recent years, PIM-based mixed-signal accelerators have been proposed as energy- and area-efficient solutions with ultra-high throughput to accelerate DNN computations. However, PIM designs are sensitive to imperfections such as noise, weight/conductance variations, and cell programming errors that substantially degrade the DNN accuracy. To address this issue, we propose a novel algorithm-hardware co-design framework called Harmonica that simultaneously avoids accuracy degradation due to imperfections, improves area utilization and execution time, and reduces energy consumption. Harmonica proposes to select imperfection-sensitive weights using an input channel-wise method and transfer them to a novel and robust digital accelerator while the main computations are performed in the analog PIM cores. Harmonica is adapted to leverage the preceding weight selection method by reducing ADC precision, employing smaller peripheral circuitry, and a hybrid quantization to optimize the design. Our comprehensive experiments show that even in the presence of imperfections as high as 50%, Harmonica reduces the accuracy degradation from 60% - 90% in designs without a protection solution (e.g., in ISAAC or SRE baselines) to 1% - 2% for different DNNs across diverse datasets. In addition, compared to the ISAAC (SRE), Harmonica improves the execution time, energy, area, power, area-efficiency, and power-efficiency by 26% (14%), 52% (40%), 28% (28%), 57% (45%), 43% (7.5×), and 91% (7.3×), respectively. By employing architecture-based differential cells, where two separated categories of crossbars are used for positive and negative weights, Harmonica outperforms ISAAC (SRE) by 75% (9.2×) and 2.65× (10.2×) in terms of area- and power-efficiency.

Graph Neural Networks (GCNs) have attracted wide attention and are applied to the real world. However, due to the ever-growing graph data with significant irregularities, off-chip communication with poor data locality has become the major bottleneck hurdling the development of GCNs. Fortunately, recent works demonstrate Resistive Random Access Memory (ReRAM) has the potential to perform inherently parallel in-situ computation of Matrix-Vector Multiplication (MVM) in the analog regime fundamentally breaking the communication bottleneck.Inspired by this observation, we propose a novel ReRAM-based GCN acceleration co-design (i.e. algorithm-hardware) framework, CoDG-ReRAM, that can deliver real-time GCN inference with high accuracy. On the algorithm side, we propose a novel model optimization pipeline that simultaneously and efficiently sparsifies and regularizes both graph and parameter matrices in GCNs and creates ReRAM-friendly models. On the hardware side, we take advantage of the software optimization results to provide a more systematic mapping scheme and in-crease computation efficiency to have an energy-efficient ReRAM-based GCN acceleration with low latency. Experimental results show that the proposed work improves performance and energy efficiency by 4× and 5.1 × respectively over SOTA ReRAM-based accelerators of GCNs with a negligible accuracy loss.

In-package DRAM cache provides a higher bandwidth than conventional memory systems. Adapting the cache management to the run-time characteristics of each application seems a promising approach improving bandwidth efficiency and performance. Regrettably, fine-grained cache block monitoring and adaptation often becomes impractical due to its significant bandwidth, performance and hardware overheads. This paper proposes a novel mechanism for monitoring cache blocks using two parameters that are adjustable at run time. We propose two low-cost counter-based mechanisms to realize the block monitors in DRAM. Moreover, we propose a novel scheduling mechanism that opportunistically transfers the counter information to the DRAM stack when the data movement overhead reaches its minimum. Our simulation results on a set of data intensive parallel applications indicate that the proposed mechanisms achieve averages of 31%, 24% performance improvements over the state-of-the-art DRAM cache architectures. System energy savings over the same baselines are 29%, 18% on average.

Aggressive scaling in integrated circuits creates new challenges such as an increase in power density, temperature, and especially process variation in designing Multiprocessor Systems-on-Chip (MPSoC). While most of the previous works attempt to mitigate the process variation effects at the system level, the eventual design still suffers from the variability of frequency and leakage power. In this paper, we propose a method calledStereothat combinesstochastic and decomposition to solve task assignment and scheduling under process variation in MPSoCs. In our previous work, we formulated a Mixed Integer Linear Programming (MILP) problem for variation-aware task assignment and scheduling to optimize energy consumption while meeting the real-time constraints. To capture the stochastic behavior of process variation, we employed a chance-constrained programming technique to turn the problem into a corresponding stochastic optimization that can be solved by typical ILP solvers. However, it had a scalability problem. To address this issue, in this work, we leverage a Logic-based Benders Decomposition (LBD) approach to improve the running time for finding an optimal solution of assignments and schedulings under process variation phenomenon). We carried out extensive experiments using Embedded System Synthesis Benchmarks Suite (E3S). The experimental results of the Stereo method evince considerable improvements compared to the baseline method in terms of performance-yield and run-time. The Stereo-based MILP method ameliorates performance-yield up to 2× and run-time by 532×. Moreover, for manifold applications, the Stereo-based LBD method archives 3.47×-91.49× run-time improvement compared to the Stereo-based MILP approach and is capable of assigning and scheduling of more than 50 tasks on 9 processors.

As the number of weight parameters in deep neural networks (DNNs) continues growing, the demand for ultra-efficient DNN accelerators has motivated research on non-traditional architectures with emerging technologies. Resistive Random-Access Memory (ReRAM) crossbar has been utilized to perform insitu matrix-vector multiplication of DNNs. DNN weight pruning techniques have also been applied to ReRAM-based mixed-signal DNN accelerators, focusing on reducing weight storage and accelerating computation. However, the existing works capture very few peripheral circuits features such as Analog to Digital converters (ADCs) during the neural network design. Unfortunately, ADCs have become the main part of power consumption and area cost of current mixed-signal accelerators, and the large overhead of these peripheral circuits is not solved efficiently. To address this problem, we propose a novel weight pruning framework for ReRAM-based mixed-signal DNN accelerators, named TINYADC, which effectively reduces the required bits for ADC resolution and hence the overall area and power consumption of the accelerator without introducing any computational inaccuracy. Compared to state-of-the-art pruning work on the ImageNet dataset, TINYADC achieves 3.5× and 2.9× power and area reduction, respectively. TINYADC framework optimizes the throughput of state-of-the-art architecture design by 29% and 40% in terms of the throughput per unit of millimeter square and watt (GOPs/s×mm2and GOPs/w), respectively.

Recent work demonstrated the promise of using resistive random access memory (ReRAM) as an emerging technology to perform inherently parallel analog domain in-situ matrix-vector multiplication—the intensive and key computation in deep neural networks (DNNs). One key problem is the weights that are signed values. However, in a ReRAM crossbar, weights are stored as conductance of the crossbar cells, and the in-situ computation assumes all cells on each crossbar column are of the same sign. The current architectures either use two ReRAM crossbars for positive and negative weights (PRIME), or add an offset to weights so that all values become positive (ISAAC). Neither solution is ideal: they either double the cost of crossbars, or incur extra offset circuity. To better address this problem, we propose FORMS, a fine-grained ReRAM-based DNN accelerator with algorithm/hardware co-design. Instead of trying to represent the positive/negative weights, our key design principle is to enforce exactly what is assumed in the in-situ computation— ensuring that all weights in the same column of a crossbar have the same sign. It naturally avoids the cost of an additional crossbar. Such polarized weights can be nicely generated using alternating direction method of multipliers (ADMM) regularized optimization during the DNN training, which can exactly enforce certain patterns in DNN weights. To achieve high accuracy, we divide the crossbar into logical sub-arrays and only enforce this property within the fine-grained sub-array columns. Crucially, the small sub-arrays provides a unique opportunity for input zero-skipping, which can significantly avoid unnecessary computations and reduce computation time. At the same time, it also makes the hardware much easier to implement and is less susceptible to non-idealities and noise than coarse-grained architectures. Putting all together, with the same optimized DNN models, FORMS achieves 1.50× and 1.93× throughput improvement in terms of $\frac{{GOPs}}{{s \times m{m^2}}}$ and $\frac{{GOPs}}{W}$ compared to ISAAC, and 1.12× ~2.4 × speed up in terms of frame per second over optimized ISAAC with almost the same power/area cost. Interestingly, FORMS optimization framework can even speed up the original ISAAC from 10.7 × up to 377.9×, reflecting the importance of software/hardware co-design optimizations.

Adapting in-package caching to run-time characteristics of user applications seems a promising approach to improve bandwidth efficiency and performance. However, fine-grained cache block monitoring and adaptation are often impractical due to the significant bandwidth and energy overheads. This paper proposes RedCache that enables fine-grained adaptation at run-time via reduced DRAM caching. Two adaptive parameters are proposed to start and stop caching for individual blocks. Architectural techniques and DRAM specific control mechanisms are proposed to alleviate overheads. Our simulation results indicate averages of 31% and 24% performance improvements over the state-of-the-art Alloy and Bear cache architectures. Respective energy savings over the same baselines are 29% and 18% on average.

Power dissipation is a significant problem limiting the performance of today's computer systems. One of the main contributors to power consumption in microprocessors is data movement in cache and memory interface. Several solutions such as low power interconnects, energy-aware data encoding, and low power signaling have been proposed to mitigate this problem. Almost all of these techniques result in a significant system performance degradation. This article examines the application of a novel technique, called STFL-DDR, for hybrid signaling on low-power DRAM interface. To keep the power consumption low, STFL-DDR employs a high-performance clock rate for transferring data on low power wires. To avoid any signal deterioration, STFL-DDR employs data encoding/decoding to prevent each wire from switching in any two consecutive cycles. STFL-DDR creates new opportunities for optimizing the energy-efficiency of DRAM systems. We compare the efficiency of STFL-DDR with the state-of-the-art methods by simulating a mix of 12 parallel benchmark applications on a muticore system. Our simulation results indicate that STFL can reduce the energy consumption of a contemporary DRAM interface by 17 percent as compared to an LPDDR baseline while achieving the throughput of a high-performance DRAM. Applying STFL to both last level cache and DRAM interface results in improving the system energy, energy-delay product, and performance by 8, 15, and 9 percent respectively. Compared with a high-performance memory interface, STFL improves the system energy and energy-delay product by 25 and 75 percent, while reaching 98 percent of the average performance of the high-performance system.

Data movement in large caches consumes a significant amount of energy in modern computer systems. Low power interfaces have been proposed to address this problem. Unfortunately, the energy-efficiency of these techniques is largely limited due to undue latency overheads of low power wires and complex coding mechanisms. This paper proposes a hybrid technique for slow-transition, fast-level (STFL) signaling that creates a balance between power and bandwidth in the last level cache interface. Combined with STFL codes, the signaling technique significantly mitigates the performance impacts of low power wires, thereby improving the energy efficiency of data movement in memory systems. When applied to the last level cache of a contemporary multicore system, STFL improves the CPU energy-delay product by 9% as compared to a voltage-frequency scaled baseline. Moreover, the proposed architecture reduces the CPU energy by 26% and achieves 98% of the performance provided by a high-performance baseline.

Computer vision algorithms are computationally expensive and difficult to implement efficiently. Field Programmable Gate Arrays (FPGA)s offer a promising direction to reduce the computation cost by exploiting hardware parallelism. However, it is difficult to translate vision algorithms to FPGA bitstream efficiently. OpenVX is an industry standard for graph-based representation of vision algorithms. It defines a set of widely used vision kernels and data structures that can be used to form a Directed Acyclic Graph (DAG) to represent a vision algorithm. This paper proposes a framework for automatic FPGA acceleration of computer vision algorithms based on OpenVX specification, called AFFIX. AFFIX receives a vision algorithm formed using the OpenVX and generates a heterogeneous CPU-FPGA implementation. AFFIX incorporates several high level and low-level optimization methods to improve the efficiency of the FPGA implementation. It provides a configurable and extensible framework that enables vision algorithm developers to quickly develop, verify and test FPGA implementations of vision algorithms. We demonstrate the effectiveness of the proposed framework via development and evaluations of multiple vision algorithms.

OpenVX is an open standard for cross platform acceleration of computer vision applications. It was created to address the challenge of implementing efficient, portable and easy to use vision processing algorithms by separating application specification and implantation. It offers a set of basic, widely used vision kernels that accelerator vendors are supposed to provide. This work presents a framework for turning a high-level OpenVX graph specification into an efficient FPGA implementation.

Over the past decade, three-dimensional die stacking technology has been considered for building large-scale in-package memory systems. In particular, in-package DRAM cache has been considered as a promising solution for high band-width and large-scale cache architectures. There are, however, significant challenges such as limited energy efficiency, costly tag management, and physical limitations for scalability that need to be effectively addressed before one can adopt in-package caches in the real-world applications. This paper proposes R-Cache, an in-package cache made by 3D die stacking of memristive memory arrays to alleviate the above mentioned challenges. Our simulation results on a set of memory intensive parallel applications indicate that R-Cache outperforms the state-of-the-art proposals for in-package caches. R-Cache improves performance by 38% and 27% over the state-of-the-art direct mapped and set associative cache architectures, respectively. Moreover, R-Cache results in averages of 40% and 27% energy reductions as compared to the direct mapped and set-associative cache systems.

Clustering is a crucial tool for analyzing data in virtually every scientific and engineering discipline. The U.S. National Academy of Sciences has recently announced “the seven giants of statistical data analysis” in which data clustering plays a central role. This report also emphasizes that more scalable solutions are required to enable time and space clustering for the future large-scale data analyses. As a result, hardware and software innovations that can significantly improve energy efficiency and performance of the data clustering techniques are necessary to make the future large-scale data analysis practical. This paper proposes a novel mechanism for computing bit-serial medians within resistive RAM arrays with no need to read out the operands from memory cells. We propose a novel four-transistor, four-memristor memory cell that enables in situ median computation within the data arrays. (If necessary, the proposed cell could be used as four ordinary one-transistor, one-memristor memory cells to store four bits of information.) The proposed hardware is used to accelerate a data clustering library using breast cancer samples, indoor localization, and the U.S. Census data sets, as well as two applications using k-means clustering. Our simulation results for the library indicate an average performance improvement of 15.5× and an energy reduction of 28.5× over a baseline CPU system. Also, we observe an overall speedup of 5.8× with an energy improvement of 14.1× over a baseline processing-in-memory accelerator. For the k-means applications, we observe speedups of 45.7× and 1.5× with respective energy improvements of 49.5× and 1.3× as compared with the CPU baseline.

In this paper, we present an efficient formal approach to check the equivalence of synthesized Register Transfer Level (RTL) against the high level specification in the presence of pipelining transformations. With the proposed equivalence checking method, fault tolerance issues when some faults happen in the designs can be formally analyzed. Equivalence checking with the specification can reason about how quickly the design can come back to normal operations when some faults happen. To increase the scalability of our proposed method, we dynamically divide the designs into several smaller parts called segments by introducing dynamic cut-points. Then we employ Modular Horner Expansion Diagram (M-HED) to check whether the specification and the implementation are equivalent or not. Our proposed method enables us to deal with the equivalence checking problem for behaviorally synthesized designs even in the presence of pipelines for nested loops. The empirical results demonstrate the efficiency and scalability of our proposed method in terms of run-time and memory usage for several large designs synthesized by a commercial behavioral synthesis tool. Average improvements in terms of the memory usage and run time in comparison with SMT- and SAT-based equivalence checking are 16.7× and 111.9×, respectively.

A large amount of time and effort must be spent to ensure the correctness of a digital design. Although many Computer Aided Design (CAD) solutions have been provided to enhance efficiency of existing debugging approaches, they are suffering from shortage of efficient automatic correction mechanisms. In this paper, we introduce an in-circuit mutation technique for correcting design bugs in digital designs. The aim of this work is reducing correction time by connecting primitive gates into inputs of 6-to-1 multiplexers in the place of potential bugs and utilizing satisfiability (SAT) engine for choosing the correct gates. The empirical results demonstrate that our proposed method can correct multiple bugs in a design by targeting gate replacements and wires exchanges efficiently. Average improvements in terms of the runtime and success rate in correction for combinational circuits in comparison with the latest the existing method are 3.4× and 11.5%, respectively. These results for sequential circuits are 3.8× and 17% respectively.

By increasing the complexity of digital systems, verification and debugging of such systems have become a major problem and economic issue. Although many computer aided design (CAD) solutions have been suggested to enhance efficiency of existing debugging approaches, they are still suffering from lack of providing a small set of potential error locations and also automatic correction mechanisms. On the other hand, the ever-growing usage of digital signal processing (DSP), computer graphics and embedded systems applications that can be modeled as polynomial computations in their datapath designs, necessitate an effective method to deal with their verification, debugging and correction. In this paper, we introduce a formal debugging approach based on static slicing and dynamic ranking methods to derive a reduced ordered set of potential error locations. In addition, to speed up finding true errors in the presence of multiple design errors, error candidates are sorted in decreasing order of their probability of being an error. After that, a mutation-based technique is employed to automatically correct bugs even in the case of multiple bugs. In order to evaluate the effectiveness of our approach, we have applied it to several industrial designs. The experimental results show that the proposed technique enables us to locate and correct even multiple bugs with high confidence in a short run time even for complex designs of up to several thousand lines of RTL code.

In this paper, we introduce a new technique that makes use of satisfiability (SAT) based debugging techniques along with a mutation-based technique to correct certain design errors in digital designs automatically. The experimental results demonstrate that our proposed method enables us to locate and correct multiple bugs by targeting gate replacements and wire exchanging within reasonable run-time and memory usage for several designs.

In this paper, we introduce a formal and scalable debugging approach to derive a reduced ordered set of design error candidates in polynomial datapath designs. To make our debugging method scalable for large designs, we utilize a Modular Horner Expansion Diagram (M-HED), which has been shown to be a scalable high level decision model. In our method, we extract data dependency graphs from the polynomial datapath designs using static slicing. Then we combine backward and forward path tracing to extract a reduced set of error candidates. In order to increase the accuracy of the method in the presence of multiple design errors, we rank the error candidates in decreasing order of their probability of being an error using a proposed priority criterion. In order to evaluate the effectiveness of our method, we have applied it to several large designs. The experimental results show that the proposed method enables us to locate even multiple errors with high accuracy in a short run time.