The hardware scheduler on NVIDIA GPUs is highly inefficient in utilizing micro-architectural hardware resources. It places blocks from the same kernel within the same GPU Streaming Multiprocessor (SM) core, resulting in a stacking colocating problem, where identical blocks are placed within the same SM core, saturating only a subset of intra-SM hardware resources while leaving others underutilized. The primary challenge in addressing this issue is that the NVIDIA hardware is closed-source, preventing us from directly modifying the hardware scheduler. To bridge the semantic gap between the resource demands of kernels and the scheduler, we introduce µ Share, which enables intra-SM scattered colocating of kernels through a non-intrusive half-plus blocksize shaping method. It shapes the blocksize of kernels to a halfplus blocksize (i.e., slightly more than half of the SM's thread capacity), scattering identical blocks of the same kernel across different SMs. It further adopts a time-shifted launching method to reduce intra-SM resource contention. Compared to state-of-the-art systems, µ Share does not require intrusive modifications to hardware or kernel code, yet it can still improve inference throughput by 26.90%-54.09% and increases low-level hardware utilization by 38.53%-61.15%.

Songtao Jiang, Yan Zhang, Yeying Jin, Zhihang Tang, Yangyang Wu, Yang Feng, Jian Wu, Zuozhu Liu. Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2025.

Training large models in parallel by GPU clusters significantly accelerates the computation in each iteration. However, the frequent collective communication for synchronizing the huge number of gradients poses a scalability challenge, whose performance gradually becomes the bottleneck as the number of workers increases. Ring-reduce is a favorable architecture since it can balance the communication and computation load among workers. In this paper, we discover that the communication resources are underutilized when using the ring-reduce synchronization method in clusters. Accordingly, Shuffle-Exchange Synchronization (SES), a novel method is proposed to improve the communication efficiency for distributed large model training. SES organizes all the worker nodes into several groups, within which they perform small-scale ring-reduce synchronizations during each iteration. To achieve better convergence performance, a gradient correction operation is integrated into SES. Experiments in 16 workers on a real-world industrial computing platform, show that SES can accelerate the large model training to 1.97× without losing model performance.

The increasing parameter size of Deep Neural Networks (DNNs) has significantly enhanced model performance. As large-scale DNN models typically require partitioning into multiple blocks for distributed training, existing research has predominantly focused on offline scheduling for individual or batched training tasks. However, the stochastic arrival of such tasks in edge networks poses a critical challenge for efficiently scheduling them in resource-constrained edge clusters. In this paper, aiming to minimize the average training completion time across all tasks, we first extract DNN operator graphs and partition them into coarse-grained subgraphs using a max-flow mincut algorithm. Then, we formulate the online scheduling problem for multiple distributed DNN training tasks as a Markov Decision Process (MDP) and propose a reinforcement learning-based (RL-based) solution. Extensive experiments comparing our method with three conventional baselines (FIFO, SJF, and Greedy) under diverse configurations show that our approach reduces the average training completion time by 12.95%, demonstrating its effectiveness in resource-constrained edge environments with dynamic workloads.