Machine learning models are distributed across multiple nodes using numerous parallelism strategies. The resulting collective communication is often on the critical path due to a lack of independent coarse-grain computation kernels available to execute. In this work, we propose fusing computation with its subsequent collective communication and leverage GPUs’ massive parallelism, along with GPU-initiated communication, to overlap communication and computation. Specifically threadblocks/workgroups (WGs) immediately communicate their results to remote GPUs after completing their computation, while other WGs within the same kernel perform computation. We developed three prototype fused operators (embedding+All-toAll, GEMV+AllReduce, and GEMM+All-to-All) to address the communication overheads in DLRM, Transformers and MoE model architectures. We expose fused kernels as new PyTorch operators, as well as extend the Triton framework to demonstrate their practicality. Our evaluations show our approach effectively overlaps communication with computations, subsequently reducing their combined execution time achieving $\mathbf{1 2 \%}$ - $\mathbf{3 1 \%}$ lower execution time across all three operators.

The pace of advancement of the top-end supercomputers historically followed an exponential curve similar to (and driven in part by) Moore's Law. Shortly after hitting the petaflop mark, the community started looking ahead to the next milestone: Exascale. However, many obstacles were already looming on the horizon, such as the slowing of Moore's Law, and others like the end of Dennard Scaling had already arrived. Anticipating significant challenges for the overall high-performance computing (HPC) community to achieve the next 1000x improvement, the U.S. Department of Energy (DOE) launched the Exascale Computing Program to enable and accelerate fundamental research across the many technologies needed to achieve exascale computing.

In multi-GPU and multi-chiplet GPU systems exhibiting NUMA behavior, information about addresses accessed by threads is crucial for various optimizations such as data/thread co-location and cache/scratchpad memory management. To make optimal decisions and avoid runtime overhead, knowledge about dynamic, potentially data-dependent access patterns should be available before kernel execution. Existing approaches require rewriting of applications or can only capture static, data-independent patterns. In this paper, we propose TAFE, a framework for accurate dynamic thread address footprint estimation of GPU applications. TAFE combines minimal static address pattern annotations with dynamic data dependency tracking to compute threadblock-specific address footprints prior to kernel launch. We propose a low-overhead software mechanism to track dynamic data-dependencies and provide an optional lightweight hardware extension to support transparent tracking.

Advancements in packaging and 3D-stacking technology have enabled novel non-uniform processing-in-memory (NUPIM) architectures integrating a network of hybrid modules each with varying sizes and proportions of compute and memory. In such systems, network congestion can have a significant impact on performance and energy-efficiency scaling. At the same time, NUPIM systems often have strict power and thermal constraints, where traditional routing/redundancy or buffering based approaches to reduce congestion cannot be applied.In this paper, we propose a low-overhead yet scalable scheme for congestion management in off-chip networks of emerging GPU-based NUPIM systems. We note that congestion in NUPIM networks exhibits a form of tree saturation in which individually congested links form congested paths, where congestion effects due to back pressure vary across paths. Additionally, memory responses can have different performance impact due to memory divergence. Our approach exploits differences in performance impact of NUPIM traffic to mitigate congestion effects. We dynamically track congested paths and divergent data to manage priorities and accelerate performance-critical traffic. We further use critical paths knowledge to dynamically manage link widths and save I/O energy.Results show that our congestion management outperforms a configuration with double the number of virtual channels and buffers indicating its effectiveness in managing congestion under stringent constraints. It improves performance by 15% (and up to 31%) over baseline and 10% (and up to 29%) against a memory distance based approach for congestion-affected benchmarks. Our scheme reduces 78% of I/O energy on average and up to 29% of system energy across all benchmarks.

General-Purpose Graphic Processing Units (GPUs) have become an integral part of heterogeneous system architectures. Ever increasing complexities have made rapid, early performance evaluation of GPU-based architectures and applications a primary design concern. Traditional cycle-accurate GPU simulators are too slow, while existing analytical or source-level estimation approaches are often inaccurate. This paper proposes a novel abstract GPU performance simulation approach that is based on flexible separation of functional and timing models, combining a fast functional execution either on existing simulators or native GPU hardware with a light, fast and accurate abstract timing model. Micro-architecture timing of individual GPU cores is abstracted through static, one-time pre-characterization of code, and only the dynamic scheduling effects are simulated. Using a native GPU for functional execution and excluding pre-characterization, our GPU simulation achieves a throughput of more than 80 MIPS. This is on average 400x faster with 4% error compared to a cycle-accurate GPU simulator for standard GPU benchmarks. Moreover, our simple timing model provides flexibility to target different GPU configurations with little or no extra effort.