The increasing complexity and diversity of hardware accelerators in modern computing systems demand flexible, low-overhead program analysis tools. We present PASTA, a low-overhead and modular Program AnalysiS Tool Framework for Accelerators. PASTA abstracts over low-level profiling APIs and diverse deep learning frameworks, offering users a unified interface to capture and analyze runtime events at multiple levels. Its extensible design enables researchers and practitioners to rapidly prototype custom tools with minimal overhead. We demonstrate the utility of PASTA by developing several analysis tools, including tools for deep learning workload characterization and UVM optimization. Through extensive evaluations on mainstream deep learning workloads tested on NVIDIA and AMD GPUs under both single- and multi-GPU scenarios, we demonstrate PASTA’s broad applicability. On NVIDIA GPUs, we further show that PASTA provides detailed performance insights with significantly lower overhead (up to 1.3×104faster) than conventional analysis tools, thanks to its GPU-accelerated backend. PASTA strikes a practical balance between usability, extensibility, and efficiency, making it well-suited for modern accelerator-based computing environments.

This paper introduces Heliostat, which enhances page translation bandwidth on GPUs by harnessing underutilized ray tracing accelerators (RTAs).While most existing studies focused on better utilizing the provided translation bandwidth, this paper introduces a new opportunity to fundamentally increase the translation bandwidth.Instead of overprovisioning the GPU memory management unit (GMMU), Heliostat repurposes the existing RTAs by leveraging the operational similarities between ray tracing and page table walks.Unlike earlier studies that utilized RTAs for certain workloads, Heliostat democratizes RTA for supporting any workloads by improving virtual memory performance.Heliostat+ optimizes Heliostat by handling predicted future address translations proactively.Heliostat outperforms baseline and two state-of-the-arts by 1.93×, 1.92×, and 1.66×.Heliostat+ further speeds up Heliostat by 1.23×.Compared to an overprovisioned comparable solution, Heliostat occupies only 1.53% of the area and consumes 5.8% of the power.

With the advancement of processor packaging technology and the looming end of Moore’s law, multi-chip-module (MCM) GPUs become a promising architecture to continue the performance scaling. However, due to the increasing concurrency, it is challenging to achieve scalable performance. In this study, we show that the limited parallelism in IOMMU is one of the critical bottlenecks and propose Barre Chord to fundamentally reduce the translation loads. By leveraging the unique GPU execution model and page mapping on MCM-GPUs, Barre translates virtual addresses in a unit of coalescing group. Once one page is translated, all the other pages within the same coalescing group can be translated with simple calculations without page table walks. Full Barre (F-Barre) further reduces translations by enabling intra-MCM translation through coalescing information sharing across GPU chiplets and contiguity-aware coalescing group expansion. With the combination of Barre and F-Barre, the Barre Chord outperforms state-of-the-art solutions by an average of 1.36× (2.09× with coalescing group expansion) with negligible area overhead (4.22% of a GPU L2 TLB).

The deep learning models (DL) are becoming bigger, easily beyond the memory capacity of a single accelerator. The recent progress in large DL training utilizes CPU memory as an extension of accelerator memory and offloads tensors to CPU memory to save accelerator memory. This solution transfers tensors between the two memories, creating a major performance bottleneck. We identify two problems during tensor transfers: (1) the coarse-grained tensor transfer creating difficulty in hiding transfer overhead, and (2) the redundant transfer that unnecessarily migrates value-unchanged bytes from CPU to accelerator. We introduce a cache coherence interconnect based on Compute Express Link (CXL) to build a cache coherence domain between CPU memory and accelerator memory. By slightly extending CXL to support an update cache-coherence protocol and avoiding unnecessary data transfers, we reduce training time by $33.7 \%$ (up to $55.4 \%$) without changing model convergence and accuracy, compared with the state-of-the-art work in DeepSpeed [62].

Graphics Processing Units (GPUs) are the accelerator of choice in a variety of application domains, because they can accelerate massively parallel workloads and can be easily programmed using general-purpose programming frameworks such as CUDA and OpenCL. Each Streaming Multiprocessor (SM) contains an L1 data cache (L1D) to exploit the locality in data accesses. L1D misses are costly for GPUs for two reasons. First, L1D misses consume a lot of energy as they need to access the L2 cache (L2) via an on-chip network and the off-chip DRAM in case of L2 misses. Second, L1D misses impose performance overhead if the GPU does not have enough active warps to hide the long memory access latency. We observe that threads running on different SMs share 55% of the data they read from the memory. Unfortunately, as the L1Ds are in the non-coherent memory domain, each SM independently fetches data from the L2 or the off-chip memory into its L1D, even though the data may be currently available in the L1D of another SM. Our goal is to service L1D read misses via other SMs, as much as possible, to cut down costly accesses to the L2 or the off-chip DRAM. To this end, we propose a new data-sharing mechanism, called Cross-Core Data Sharing (CCDS) . CCDS employs a predictor to estimate whether the required cache block exists in another SM. If the block is predicted to exist in another SM’s L1D, then CCDS fetches the data from the L1D that contain the block. Our experiments on a suite of 26 workloads show that CCDS improves average energy and performance by 1.30× and 1.20×, respectively, compared to the baseline GPU. Compared to the state-of-the-art data-sharing mechanism, CCDS improves average energy and performance by 1.37× and 1.11×, respectively.

Memory capacity is a major bottleneck for training deep neural networks (DNN). Heterogeneous memory (HM) combining fast and slow memories provides a promising direction to increase memory capacity. However, HM imposes challenges on tensor migration and allocation for high performance DNN training. Prior work heavily relies on DNN domain knowledge, unnecessarily causes tensor migration due to page-level false sharing, and wastes fast memory space. We present Sentinel, a software runtime system that automatically optimizes tensor management on HM. Sentinel uses dynamic profiling, and coordinates operating system (OS) and runtime-level profiling to bridge the semantic gap between OS and applications, which enables tensor-level profiling. This profiling enables co-allocating tensors with similar lifetime and memory access frequency into the same pages. Such fine-grained profiling and tensor collocation avoids unnecessary data movement, improves tensor movement efficiency, and enables larger batch training because of saving in fast memory space. Sentinel reduces fast memory consumption by 80% while retaining comparable performance to fast memory-only system; Sentinel consistently outperforms a state-of-the-art solution on CPU by 37% and two state-of-the-art solutions on GPU by 2x and 21% respectively in training throughput.

The Register File (RF) in GPUs is a critical structure that maintains the state for thousands of threads that support the GPU processing model. The RF organization substantially affects the overall performance and the energy efficiency of a GPU. For example, the frequent accesses to the RF consume a substantial amount of the dynamic energy, and port contention due to limited ports on operand collectors and register file banks affect performance as register operations are serialized. We present CORF, a compiler-assisted Coalescing Operand Register File which performs register coalescing by combining reads to multiple registers required by a single instruction, into a single physical read. To enable register coalescing, CORF utilizes register packing to co-locate narrow-width operands in the same physical register. CORF uses compiler hints to identify which register pairs are commonly accessed together. CORF saves dynamic energy by reducing the number of physical register file accesses, and improves performance by combining read operations, as well as by reducing pressure on the register file. To increase the coalescing opportunities, we re-architect the physical register file to allow coalescing reads across different physical registers that reside in mutually exclusive sub-banks; we call this design CORF++. The compiler analysis for register allocation for CORF++ becomes a form of graph coloring called the bipartite edge frustration problem. CORF++ reduces the dynamic energy of the RF by 17%, and improves IPC by 9%.

Cluster tools are automated robotic manufacturing systems andwidely used in semiconductor industry. As the structure of semi-conductor chips becomes more complicated, the complexity of fab-ricating process and the number of fabricating steps are increased.Therefore, the throughput of cluster tool becomes more difficult topredict. The throughput of a cluster tool is defined as the numberof wafers processed in an hour. The throughput of cluster tools hasnonlinear relationship with operating parameters, such as recipetime, cleaning (W ACT M) time, and look-ahead time (JIT1). Existingthroughput models have limitations that only provide the worst andthe best throughput boundaries and have exponentially increasingcomplexity when considering more resources and scheduling steps.Instead, we propose to use machine learning algorithms for clus-ter tool throughput prediction. To understand the nonlinearity ofthroughput of cluster tool and identify the best algorithm that rep-resents the behavior of cluster tool, we evaluated various machinelearning algorithms as well as deep neural networks (DNN). AsDNN performance varies significantly depending on the architec-ture of the network, we explored both depth and width dimensionsof architecture; going deeper (more layers) and going wider (moreneurons per layers). Our evaluation shows that the throughput ofcluster tools with multiple parallel PMs with dual transfer cham-ber arms, buffered AirLocks, and cleaning after recipe can be wellpredicted with deeper DNNs than wider DNNs.

Deep neural networks (DNNs) have been proving the effectiveness in various computing fields. To provide more efficient computing platforms for DNN applications, it is essential to have evaluation environments that include assorted benchmark workloads. Though a few DNN benchmark suites have been recently released, most of them require to install proprietary DNN libraries or resource-intensive DNN frameworks, which are hard to run on resource-limited mobile platforms or architecture simulators. To provide a more scalable evaluation environment, we propose a new DNN benchmark suite that can run on any platform that supports CUDA and OpenCL. The proposed benchmark suite includes the most widely used five convolution neural networks and two recurrent neural networks. We provide architectural statistics of these networks while running them on an architecture simulator, a server- and a mobile-GPU, and a mobile FPGA.

Albeit GPUs are supposed to be tolerant to long latency of data fetch operation, we observe that L1 cache misses occur in a bursty manner for many memory-intensive applications. This in turn leads to severe contentions in GPU memory hierarchy, and thus stalls execution pipeline for many cycles as all warps end up waiting for their memory requests to be serviced by L1 cache. To spread such bursty L1 cache misses, we propose CTA-Aware Prefetcher and Scheduler (CAPS) consisting of a thread group-aware prefetcher and a prefetch-aware warp scheduler for GPUs. GPU kernels group threads into cooperative thread arrays (CTAs). Each thread typically uses its thread index and its associated CTA index to identify the data that it operates on. The starting base address accessed by the first warp in a CTA is difficult to predict since that starting address is a complex function of thread index and CTA index and also depends on how the programmer distributes input data across CTAs. But threads within each CTA exhibit stride accesses. Hence, if the base address of each CTA can be computed early, it is possible to accurately predict prefetch addresses for threads within a CTA. To compute the base address of each CTA, a leading warp is used from each CTA. The leading warp is executed early by pairing it with warps from currently executing leading CTA. The warps in the leading CTA are used to compute the stride value. The stride value is then combined with base addresses computed from the leading warp of each CTA to prefetch the data for all the trailing warps in the trailing CTAs. CAPS allows prefetch requests to be issued sufficiently ahead of time before the demand requests, effectively reorganizing warp executions to quickly detect the base address of each CTA and stride per load. CAPS predicts addresses with over 97% accuracy and is able to improve GPU performance by 8% on average with up to 27% for a wide range of GPU applications.

GPU adoption for general purpose computing has been accelerating. To support a large number of concurrently active threads, GPUs are provisioned with a very large register file (RF). The RF power consumption is a critical concern. One option to reduce the power consumption dramatically is to use near-threshold voltage(NTV) to operate the RF. However, operating MOSFET devices at NTV is fraught with stability and reliability concerns. The adoption of FinFET devices in chip industry is providing a promising path to operate the RF at NTV while satisfactorily tackling the stability and reliability concerns. However, the fundamental problem of NTV operation, namely slow access latency, remains. To tackle this challenge in this paper we propose to build a partitioned RF using FinFET technology. The partitioned RF design exploits our observation that applications exhibit strong preference to utilize a small subset of their registers. One way to exploit this behavior is to cache the RF content as has been proposed in recent works. However, caching leads to unnecessary area overheads since a fraction of the RF must be replicated. Furthermore, we show that caching is not efficient as we increase the number of issued instructions per cycle, which is the expected trend in GPU designs. The proposed partitioned RF splits the registers into two partitions: the highly accessed registers are stored in a small RF that switches between high and low power modes. We use the FinFET's back gate control to provide low overhead switching between the two power modes. The remaining registers are stored in a large RF partition that always operates at NTV. The assignment of the registers to the two partitions will be based on statistics collected by the a hybrid profiling technique that combines the compiler based profiling and the pilot warp profiling technique proposed in this paper. The partitioned FinFET RF is able to save 39% and 54% of the RF leakage and the dynamic energy, respectively, and suffers less than 2% performance overhead.

GPU design trends show that the register file size will continue to increase to enable even more thread level parallelism. As a result register file consumes a large fraction of the total GPU chip power. This paper explores register file data compression for GPUs to improve power efficiency. Compression reduces the width of the register file read and write operations, which in turn reduces dynamic power. This work is motivated by the observation that the register values of threads within the same warp are similar, namely the arithmetic differences between two successive thread registers is small. Compression exploits the value similarity by removing data redundancy of register values. Without decompressing operand values some instructions can be processed inside register file, which enables to further save energy by minimizing data movement and processing in power hungry main execution unit. Evaluation results show that the proposed techniques save 25 percent of the total register file energy consumption and 21 percent of the total execution unit energy consumption with negligible performance impact.

As technology scales, GPUs are forecasted to incorporate an ever-increasing amount of computing resources to support thread-level parallelism. But even with the best effort, exposing massive thread-level parallelism from a single GPU kernel, particularly from general purpose applications, is going to be a difficult challenge. In some cases, even if there is sufficient thread-level parallelism in a kernel, there may not be enough available memory bandwidth to support such massive concurrent thread execution. Hence, GPU resources may be underutilized as more general purpose applications are ported to execute on GPUs. In this paper, we explore multiprogramming GPUs as a way to resolve the resource underutilization issue. There is a growing hardware support for multiprogramming on GPUs. Hyper-Q has been introduced in the Kepler architecture which enables multiple kernels to be invoked via tens of hardware queue streams. Spatial multitasking has been proposed to partition GPU resources across multiple kernels. But the partitioning is done at the coarse granularity of streaming multiprocessors (SMs) where each kernel is assigned to a subset of SMs. In this paper, we advocate for partitioning a single SM across multiple kernels, which we term as intra-SM slicing. We explore various intra-SM slicing strategies that slice resources within each SM to concurrently run multiple kernels on the SM. Our results show that there is not one intra-SM slicing strategy that derives the best performance for all application pairs. We propose Warped-Slicer, a dynamic intra-SM slicing strategy that uses an analytical method for calculating the SM resource partitioning across different kernels that maximizes performance. The model relies on a set of short online profile runs to determine how each kernel's performance varies as more thread blocks from each kernel are assigned to an SM. The model takes into account the interference effect of shared resource usage across multiple kernels. The model is also computationally efficient and can determine the resource partitioning quickly to enable dynamic decision making as new kernels enter the system. We demonstrate that the proposed Warped-Slicer approach improves performance by 23% over the baseline multiprogramming approach with minimal hardware overhead.

Graphics processing units (GPUs) are now the dominant computing fabric within many supercomputers. As such many mission critical applications run on GPUs, which demand stringent reliability and computational correctness guarantees from GPUs. Prior approaches to GPU reliability have tackled solely either error detection, or error correction assuming error detection is already present. In this paper we present Warped Redundant Execution (Warped-RE), a unified framework that is capable of detecting and then correcting transient and nontransient errors in the GPU execution lanes. Our work exploits two critical properties of applications running on GPUs. First, we observe that neighboring execution lanes in GPUs may operate on the same values. Thus when neighboring lanes execute the same instruction using the same values then these lanes provide inherent DMR (dual modular redundancy) or even inherent TMR (triple modular redundancy) opportunities. The second property we exploit is that due to insufficient parallelism or due to branch divergence, applications do not fully utilize all the available execution lanes. In this case it is possible to force DMR or TMR on unused execution lanes, when inherent redundancy is insufficient. During error-free execution, Warped-RE uses a combination of inherent and forced DMR to guarantee that every thread computation within every warp instruction will be verified. When an error is detected in a warp instruction, the instruction is re-executed in TMR mode in order to correct the error and identify execution lanes with potential non-transient errors. Our evaluations show 8.4% and 29% average performance overhead during the DMR and TMR operation modes, respectively. Compared to traditional DMR and TMR, Warped-RE reduces the power overhead by 42% and 40%, respectively.

This paper presents Warped-Compression, a warp-level register compression scheme for reducing GPU power consumption. This work is motivated by the observation that the register values of threads within the same warp are similar, namely the arithmetic differences between two successive thread registers is small. Removing data redundancy of register values through register compression reduces the effective register width, thereby enabling power reduction opportunities. GPU register files are huge as they are necessary to keep concurrent execution contexts and to enable fast context switching. As a result register file consumes a large fraction of the total GPU chip power. GPU design trends show that the register file size will continue to increase to enable even more thread level parallelism. To reduce register file data redundancy warped-compression uses low-cost and implementation-efficient base-delta-immediate (BDI) compression scheme, that takes advantage of banked register file organization used in GPUs. Since threads within a warp write values with strong similarity, BDI can quickly compress and decompress by selecting either a single register, or one of the register banks, as the primary base and then computing delta values of all the other registers, or banks. Warped-compression can be used to reduce both dynamic and leakage power. By compressing register values, each warp-level register access activates fewer register banks, which leads to reduction in dynamic power. When fewer banks are used to store the register content, leakage power can be reduced by power gating the unused banks. Evaluation results show that register compression saves 25% of the total register file power consumption.

To support massive number of parallel thread contexts, Graphics Processing Units (GPUs) use a huge register file, which is responsible for a large fraction of GPU's total power and area. The conventional belief is that a large register file is inevitable for accommodating more parallel thread contexts, and technology scaling makes it feasible to incorporate ever increasing size of register file. In this paper, we demonstrate that the register file size need not be large to accommodate more threads context. We first characterize the useful lifetime of a register and show that register lifetimes vary drastically across various registers that are allocated to a kernel. While some registers are alive for the entire duration of the kernel execution, some registers have a short lifespan. We propose GPU register file virtualization that allows multiple warps to share physical registers. Since warps may be scheduled for execution at different points in time, we propose to proactively release dead registers from one warp and re-allocate them to a different warp that may occur later in time, thereby reducing the needless demand for physical registers. By using register virtualization, we shrink the architected register space to a smaller physical register space. By under-provisioning the physical register file to be smaller than the architected register file we reduce dynamic and static power consumption. We then develop a new register throttling mechanism to run applications that exceed the size of the under-provisioned register file without any deadlock. Our evaluation shows that even after halving the architected register file size using our proposed GPU register file virtualization applications run successfully with negligible performance overhead.

Die-stacked (3D) DRAM is a promising memory architecture to satisfy the high bandwidth and low latency needs of many computing systems. However, with technology scaling, memory devices are expected to experience significant increase in single and multi-bit errors. 3D DRAM will have the added burden of protecting against single through-silicon-via (TSV) failures, which translate into multiple bit errors in a single cache line, as well as multiple TSV failures that may lead to an entire channel failure. To exploit the wide interface capability of 3D DRAM, large chunks of data are laid out contiguously in a single channel and an entire cache line is sourced from a single channel. Conventional approaches such as ECC DIMM and chipkill-correct are inefficient in the context of 3D DRAM because they spread data across multiple DRAM layers to protect against failures and also place restrictions on the number of memory layers that must be protected together. This paper proposes a two-level error correction technique while taking into account 3D DRAM's unique organization. First, we propose a new symbol-based ECC layout to cover 3D DRAM specific errors as well as various well-known DRAM failure modes. Then, an XOR based correction code (XCC) is used to correct the multi-bit errors that are not correctable by the symbol based ECC. To take advantage of 3D DRAM's channel level parallelism, a permutation-based ECC placement is used. As an optimization, an ECC cache and decoupled XCC update are used for improving read and write performance without compromising reliability. The proposed approaches effectively reduce the FIT rate with almost negligible performance overhead.

General purpose graphics processing units (GPGPUs) are feature rich GPUs that provide general purpose computing ability with massive number of parallel threads. The massive parallelism combined with programmability made GPGPUs the most attractive choice in supercomputing centers. Unsurprisingly, most of the GPGPU-based studies have been focusing on performance improvement leveraging GPGPU's high degree of parallelism. However, for many scientific applications that commonly run on supercomputers, program correctness is as important as performance. Few soft or hard errors could lead to corrupt results and can potentially waste days or even months of computing effort. In this research we exploit unique architectural characteristics of GPGPUs to propose a light weight error detection method, called Warped Dual Modular Redundancy (Warped-DMR). Warped-DMR detects errors in computation by relying on opportunistic spatial and temporal dual-modular execution of code. Warped-DMR is light weight because it exploits the underutilized parallelism in GPGPU computing for error detection. Error detection spans both within a warp as well as between warps, called intra-warp and inter-warp DMR, respectively. Warped-DMR achieves 96% error coverage while incurring a worst-case 16% performance overhead without extra execution units or programmer's effort.

Exact inference is a key problem in exploring probabilistic graphical models. The computational complexity of inference increases dramatically with the parameters of the graphical model. To achieve scalability over hundreds of threads remains a fundamental challenge. In this paper, we use a lightweight scheduler hosted by the CPU to allocate cliques in junction trees to the GPGPU at run time. The scheduler merges multiple small cliques or splits large cliques dynamically so as to maximize the utilization of the GPGPU resources. We implement node level primitives on the GPGPU to process the cliques assigned by the CPU. We propose a conflict free potential table organization and an efficient data layout for coalescing memory accesses. In addition, we develop a double buffering based asynchronous data transfer between CPU and GPGPU to overlap clique processing on the GPGPU with data transfer and scheduling activities. Our implementation achieved 30× speedup compared with state-of-the-art multicore processors.

Load balancing has been known as an essential feature for enhancing the performance of distributed systems. For embedded systems, however, this is not always true since load balancing leads to lavish power consumption by fully utilizing all the embedded cores even for a small number of tasks. Furthermore, the previously proposed load unbalancing strategies do not concern much about the characteristics of the embedded system's real workload. In this paper, to resolve this problem, we propose a novel load unbalancing strategy based on the task characteristics: periodic and aperiodic. In the proposed strategy, the periodic tasks that are more likely to be executed repeatedly are concentrated on the minimum number of cores, whereas the aperiodic tasks that are not likely to occur again soon are distributed to the maximum number of cores. The experimental results on an ARM11MPCore test chip show that the proposed strategy reduces power consumption and mean waiting time of the aperiodic tasks by up to 26 percent and 82 percent, respectively, compared to the load balancing strategy. As compared to the aggressive load unbalancing strategy, the proposed strategy also reduces mean waiting time of the aperiodic tasks by 92 percent with similar power efficiency.