The arrival of 100K+ GPU clusters marks a new frontier in AI infrastructure. Standard communication stack meets new challenges as physical topologies span multiple datacenter buildings, introducing high bandwidth-delay product links where latency increases by up to 30× compared to intra-rack traffic. Furthermore, the transition toward Mixture-of-Experts architectures generating bursty all-to-all patterns that create transient congestion hotspots. These constraints, combined with an operational environment where hardware failures shift from anomalies to frequent occurrences, renders traditionally lightweight operations like initialization and resource management challenging.

Meta maintains additional capacity within its infrastructure to ensure high availability for business workloads, accommodating user growth, temporal traffic variations, and unforeseen regional failures. However, this strategic choice inherently leads to underutilization of resources. We employ oversubscription as an effective strategy to mitigate infrastructure underutilization.