Modern GPUs feature specialized hardware units that enable high-performance, asynchronous dataflow execution. However, the conventional SIMT programming model is fundamentally misaligned with this task-parallel hardware, creating a significant programmability gap. While hardware-level warp specialization is the key to unlocking peak performance, it forces developers to manually orchestrate complex, low-level communication and software pipelines—a process that is labor-intensive, error-prone, and unsustainable. To address this challenge, we present Tawa, an automated compiler that systematically generates high-performance, warp-specialized code from a high-level, tile-based program. Central to our approach is a novel IR abstraction, asynchronous references (aref), which expresses warp-level communication without exposing low-level hardware details. Using this abstraction, Tawa automatically partitions programs into producer-consumer roles and manages the intricate dataflow pipeline, relieving developers of invasive kernel rewriting. Evaluation on NVIDIA H100 GPUs across representative LLM kernels shows that Tawa delivers high hardware utilization, achieving up to 1.1 × speedup over highly optimized cuBLAS GEMM kernels. For attention workloads, Tawa attains 1.2× speedup over Triton and matches the performance of the hand-optimized CUTLASS C++ FlashAttention-3 kernel with far less programming effort.

Compute-in-SRAM architectures offer a promising approach to achieving higher performance and energy efficiency across a range of data-intensive applications.However, prior evaluations have largely relied on simulators or small prototypes, limiting the understanding of their real-world potential.In this work, we present a comprehensive performance and energy characterization of a commercial compute-in-SRAM device, the GSI APU, under realistic workloads.We compare the GSI APU against established architectures, including CPUs and GPUs, to quantify its energy efficiency and performance potential.We introduce an analytical framework for general-purpose compute-in-SRAM devices that reveals fundamental optimization principles by modeling performance trade-offs, thereby guiding program optimizations.Exploiting the fine-grained parallelism of tightly integrated memory-compute architectures requires careful data management.We address this by proposing three optimizations: communicationaware reduction mapping, coalesced DMA, and broadcast-friendly data layouts.When applied to retrieval-augmented generation (RAG) over large corpora (10GB-200GB), these optimizations enable our compute-in-SRAM system to accelerate retrieval by 4.8×-6.6×over an optimized CPU baseline, improving end-to-end RAG latency by 1.1×-1.8×.The shared off-chip memory bandwidth is modeled using a simulated HBM, while all other components are measured on the real compute-in-SRAM device.Critically, this system matches the performance of an NVIDIA A6000 GPU for RAG while being significantly more energy-efficient (54.4×-117.9×reduction).These findings validate the viability of compute-in-SRAM for complex, real-world applications and provide guidance for advancing the technology.

Recent advancements in large language models (LLMs) boasting billions of parameters have generated a significant demand for efficient deployment in inference workloads. While hardware accelerators for Transformer-based models have been extensively studied, the majority of existing approaches rely on temporal architectures that reuse hardware units for different network layers and operators. However, these methods often encounter challenges in achieving low latency due to considerable memory access overhead. This article investigates the feasibility and potential of model-specific spatial acceleration for LLM inference on field-programmable gate arrays (FPGAs). Our approach involves the specialization of distinct hardware units for specific operators or layers, facilitating direct communication between them through a dataflow architecture while minimizing off-chip memory accesses. We introduce a comprehensive analytical model for estimating the performance of a spatial LLM accelerator, taking into account the on-chip compute and memory resources available on an FPGA. This model can be extended to multi-FPGA settings for distributed inference. Through our analysis, we can identify the most effective parallelization and buffering schemes for the accelerator and, crucially, determine the scenarios in which FPGA-based spatial acceleration can outperform its GPU-based counterpart. To enable more productive implementations of an LLM model on FPGAs, we further provide a library of high-level synthesis (HLS) kernels that are composable and reusable. This library will be made available as open-source. To validate the effectiveness of both our analytical model and HLS library, we have implemented Bidirectional Encoder Representations from Transformers (BERT) and Generative Pre-trained Transformers (GPT2) on an AMD Xilinx Alveo U280 FPGA device. Experimental results demonstrate our approach can achieve up to 13.4× speedup when compared to previous FPGA-based accelerators for the BERT model. For GPT generative inference, we attain a 2.2× speedup compared to Design for Excellence, an FPGA overlay, in the prefill stage, while achieving a 1.9× speedup and a 5.7× improvement in energy efficiency compared to the NVIDIA A100 GPU in the decode stage.

Recent years have seen an increase in the development of large deep learning (DL) models, which makes training efficiency crucial. Common practice is struggling with the trade-off between usability and performance. On one hand, DL frameworks such as PyTorch use dynamic graphs to facilitate model developers at a price of sub-optimal model training performance. On the other hand, practitioners propose various approaches to improving the training efficiency by sacrificing some of the flexibility, ranging from making the graph static for more thorough optimization (e.g., XLA) to customizing optimization towards large-scale distributed training (e.g., DeepSpeed and Megatron-LM).

Recent advancements in large language models (LLMs) have generated significant demands for efficient deployment in inference workloads. Most existing approaches rely on temporal architectures that reuse hardware units for different network layers and operators. However, these methods often encounter challenges in achieving low latency due to considerable memory access overhead. This paper investigates the feasibility and potential of model-specific spatial acceleration for LLM inference on FPGAs. Our approach involves the specialization of distinct hardware units for specific operators, facilitating direct communication between them through a dataflow architecture while minimizing off-chip memory accesses. We introduce a comprehensive analytical model for estimating the performance of a spatial LLM accelerator, considering the available hardware resources on an FPGA. Through our analysis, we can determine the scenarios in which FPGA-based spatial acceleration can outperform its GPU-based counterpart. To enable more productive implementations of an LLM model on FPGAs, we further provide a library of high-level synthesis (HLS) kernels that are composable and reusable. To validate the effectiveness of both our analytical model and HLS library, we have implemented BERT and GPT2 on an AMD Alveo U280 FPGA. Experimental results demonstrate our approach can achieve up to 16.1x speedup when compared to previous FPGA-based accelerators for the BERT model. For GPT generative inference, we attain a 2.2x speedup compared to DFX, an FPGA overlay, in the prefill stage, while achieving a 1.9x speedup and a 5.7x improvement in energy efficiency compared to the NVIDIA A100 GPU in the decode stage.

High-level synthesis (HLS) can greatly facilitate the description of complex hardware implementations, by raising the level of abstraction up to a classical imperative language such as C/C++, usually augmented with vendor-specific pragmas and APIs. Despite productivity improvements, attaining high performance for the final design remains a challenge, and higher-level tools like source-to-source compilers have been developed to generate programs targeting HLS toolchains. These tools may generate highly complex HLS-ready C/C++ code, reducing the programming effort and enabling critical optimizations. However, whether these HLS-friendly programs are produced by a human or a tool, validating their correctness or exposing bugs otherwise remains a fundamental challenge. In this work we target the problem of efficiently checking the semantics equivalence between two programs written in C/C++ as a means to ensuring the correctness of the description provided to the HLS toolchain, by proving an optimized code version fully preserves the semantics of the unoptimized one. We introduce a novel formal verification approach that combines concrete and abstract interpretation with a hybrid symbolic analysis. Notably, our approach is mostly agnostic to how control-flow, data storage, and dataflow are implemented in the two programs. It can prove equivalence under complex bufferization and loop/syntax transformations, for a rich class of programs with statically interpretable control-flow. We present our techniques and their complete end-to-end implementation, demonstrating how our system can verify the correctness of highly complex programs generated by source-to-source compilers for HLS, and detect bugs that may elude co-simulation.

Special-purpose hardware accelerators are increasingly pivotal for sustaining performance improvements in emerging applications, especially as the benefits of technology scaling continue to diminish. However, designers currently lack effective tools and methodologies to construct complex, high-performance accelerator architectures in a productive manner. Existing high-level synthesis (HLS) tools often require intrusive sourcelevel changes to attain satisfactory quality of results. Despite the introduction of several new accelerator design languages (ADLs) aiming to enhance or replace HLS, their advantages are more evident in relatively simple applications with a single kernel. Existing ADLs prove less effective for realistic hierarchical designs with multiple kernels, even if the design hierarchy is flattened. In this paper, we introduce Allo, a composable programming model for efficient spatial accelerator design. Allo decouples hardware customizations, including compute, memory, communication, and data type from algorithm specification, and encapsulates them as a set of customization primitives. Allo preserves the hierarchical structure of an input program by combining customizations from different functions in a bottomup, type-safe manner. This approach facilitates holistic optimizations that span across function boundaries. We conduct comprehensive experiments on commonly-used HLS benchmarks and several realistic deep learning models. Our evaluation shows that Allo can outperform state-of-the-art HLS tools and ADLs on all test cases in the PolyBench. For the GPT2 model, the inference latency of the Allo generated accelerator is 1.7× faster than the NVIDIA A100 GPU with 5.4× higher energy efficiency, demonstrating the capability of Allo to handle large-scale designs.

The past decade has witnessed increasing adoption of high-level synthesis (HLS) to implement specialized hardware accelerators targeting either FPGAs or ASICs. However, current HLS programming models entangle algorithm specifications with hardware customization techniques, which lowers both the productivity and portability of the accelerator design. To tackle this problem, recent efforts such as HeteroCL propose to decouple algorithm definition from essential hardware customization techniques in compute, data type, and memory, increasing productivity, portability, and performance.

To achieve high performance with FPGA-equipped heterogeneous compute systems, it is crucial to co-optimize data placement and compute scheduling to maximize data reuse and bandwidth utilization for both on- and off-chip memory accesses. However, optimizing the data placement for FPGA accelerators is a complex task. One must acquire in-depth knowledge of the target FPGA device and its associated memory system in order to apply a set of advanced optimizations. Even with the latest high-level synthesis (HLS) tools, programmers often have to insert many low-level vendor-specific pragmas and substantially restructure the algorithmic code so that the right data are accessed at the right loop level using the right communication schemes. These code changes can significantly compromise the composability and portability of the original program. To address these challenges, we propose HeteroFlow, an FPGA accelerator programming model that decouples the algorithm specification from optimizations related to orchestrating the placement of data across a customized memory hierarchy. Specifically, we introduce a new primitive named .to(), which provides a unified programming interface for specifying data placement optimizations at different levels of granularity: (1) coarse-grained data placement between host and accelerator, (2) medium-grained kernel-level data placement within an accelerator, and (3) fine-grained data placement within a kernel. We build HeteroFlow on top of the open-source HeteroCL DSL and compilation framework. Experimental results on a set of realistic benchmarks show that, programs written in HeteroFlow can match the performance of extensively optimized manual HLS design with much fewer lines of code.

Binary neural networks (BNNs) have 1-bit weights and activations. Such networks are well suited for FPGAs, as their dominant computations are bitwise arithmetic and the memory requirement is also significantly reduced. However, compared to start-of-the-art compact convolutional neural network (CNN) models, BNNs tend to produce a much lower accuracy on realistic datasets such as ImageNet. In addition, the input layer of BNNs has gradually become a major compute bottleneck, because it is conventionally excluded from binarization to avoid a large accuracy loss.

As a large number of emerging graph applications spread across different domains, the need for processing massive concurrent graph jobs (CGJs) is increasing. However, existing graph processing systems designed for a single job cannot efficiently tackle multiple CGJs, where they suffer from interfering memory access patterns and inefficient property management. In this paper, we introduce Krill, a compiler and runtime system for processing concurrent graph jobs. We propose an SAP model, which decouples graph structure, algorithm, and property. In the compiler, we propose leveraging the property buffer to easily write and manage property data. In the runtime system, we propose a novel technique named graph kernel fusion to reduce memory accesses, which fuses all the jobs and processes them as a whole. Experimental results show our system significantly reduces the number of memory accesses for CGJs by more than 6x compared with the baseline, and achieves up to 6.76x speedup with 3.84x shorter response latency than GraphM, the state-of-the-art concurrent graph processing system.

High-level synthesis (HLS) is important for compiling an application design onto field-programmable gate array (FPGA) but still faces challenges of balancing scalability and quality of results in the scheduling process. In this article, we propose an entropy-directed scheduling (EDS) algorithm that efficiently generates high-quality schedules for FPGA HLS. This article is novel in three ways. First, we make the first attempt to adopt entropy in the scheduling of HLS, which is an intuitive and robust measurement with lots of good analytic properties. Second, we creatively leverage the maximum entropy principle to describe the scheduling process, which is proved equivalent to the optimal solution in some particular cases. Third, we make EDS automatically analyze the input graph structure and leverage a three-stage scheduling process to obtain high-quality results. As a result, EDS has the lowest time complexity among existing scheduling algorithms and is flexible to solve both latency- and resource-constrained problems while satisfying other constraints. The experimental results show that for latency-constrained scheduling problem, EDS reduces up to 68% resource usage and is 294× faster than the force-directed scheduling algorithm. For resource-constrained scheduling problem, EDS obtains near-optimal solutions with an average speedup of 16 410× compared with ILP. To our best knowledge, this is the first EDS algorithm for FPGA HLS.

As the most important stage in high-level synthesis (HLS), scheduling mostly relies on heuristic algorithms due to their speed, flexibility, and scalability. However, designing heuristics easily involves human bias, which makes the scheduling unpredictable in some specific cases. In this paper, we propose a deep-reinforcement-learning (Deep-RL) based scheduler for HLS. It maximumly reduces the human involvement and learns to schedule by itself. Firstly, we introduce a novel state and action representation for constrained scheduling problems, which is the foundation of the learning task. Secondly, we use a training pipeline to train the policy network. Supervised learning is used to initialize the weight of the network, and reinforcement learning is used to improve the performance, which makes the Deep-RL based scheduler practical for HLS. Finally, we compare our scheduler with the ASAP schedule and the optimal ILP schedule. Experimental results show our scheduler can reduce up to 74% resource usage compared with the original ASAP schedule, and the gap between the optimal solution is small. Notably, this is the first work leveraging reinforcement learning in HLS and has great potential to be integrated into different HLS systems.

As the most critical stage in FPGA HLS, scheduling depends heavily on heuristics due to their speed, flexibility, and scalability. However, designing heuristics easily involves human bias, which makes scheduling unpredictable in some specific cases. To solve the problem, we propose an efficient deep reinforcement learning (Deep-RL) based scheduler for FPGA HLS. It has the potential to reduce the human involvement maximumly and learn to schedule by itself. The proposed scheduler consists of three steps. First, we design a novel state and action representation for constrained scheduling problems, which is the foundation of the learning task. Then, we leverage a training pipeline to train the policy network. Specifically, supervised learning is used to initialize the weights of the network and reinforcement learning is used to improve the performance, both of which make the Deep-RL based scheduler practical for HLS. At last, we compare our scheduler with the ASAP schedule and the optimal ILP schedule. Experimental results show that the proposed scheduler can reduce up to 74% resource usage compared with the original ASAP schedule, and the gap between the optimal solution is very small. Notably, this is the first work leveraging reinforcement learning in HLS and has great potential to be integrated into different HLS systems.