The hardware scheduler on NVIDIA GPUs is highly inefficient in utilizing micro-architectural hardware resources. It places blocks from the same kernel within the same GPU Streaming Multiprocessor (SM) core, resulting in a stacking colocating problem, where identical blocks are placed within the same SM core, saturating only a subset of intra-SM hardware resources while leaving others underutilized. The primary challenge in addressing this issue is that the NVIDIA hardware is closed-source, preventing us from directly modifying the hardware scheduler. To bridge the semantic gap between the resource demands of kernels and the scheduler, we introduce µ Share, which enables intra-SM scattered colocating of kernels through a non-intrusive half-plus blocksize shaping method. It shapes the blocksize of kernels to a halfplus blocksize (i.e., slightly more than half of the SM's thread capacity), scattering identical blocks of the same kernel across different SMs. It further adopts a time-shifted launching method to reduce intra-SM resource contention. Compared to state-of-the-art systems, µ Share does not require intrusive modifications to hardware or kernel code, yet it can still improve inference throughput by 26.90%-54.09% and increases low-level hardware utilization by 38.53%-61.15%.

Computing systems always face a “resource allocation dilemma” that shows the great difficulties in trading off resource efficiency for tail latency, due to the internal uncertainty of cluster status and execution behavior. Inspired by the imaginary “Maxwell’s demon” in thermodynamics who can reduce the uncertainty through a per-gas molecule-level control policy, we consider the “one-to-one mapping” feature of serverless computing and build a novel resource allocator, named Maxwell, that can achieve low tail latency and high resource efficiency in serverless simultaneously. Like the “Maxwell’s demon Maxwell is able to optimize the resource allocation for every request. It observes the state of each request and makes decisions about the minimum resource allocation through a reinforcement learning predictor. As the per-request-grained control incurs significant overhead, we further design a pipeline for avoiding the accumulated effect on a workflow. Experimental results show that Maxwell not only saves up to 31% CPU resources but also reduces the standard deviation of latency by 1.9×. Its time overhead is negligible and the resource overhead is also limited when the query per second $\leq$500.