Today, large AI training jobs crash for a broad range of reasons and recovery often involves significant amounts of human intervention. At Meta, we observed that 5-20% of our job failures are actually due to kernel bugs in NIC drivers. Unlike other class of bugs, these are challenging to diagnose because we lack visibility into this aspect of the kernel.

The arrival of 100K+ GPU clusters marks a new frontier in AI infrastructure. Standard communication stack meets new challenges as physical topologies span multiple datacenter buildings, introducing high bandwidth-delay product links where latency increases by up to 30× compared to intra-rack traffic. Furthermore, the transition toward Mixture-of-Experts architectures generating bursty all-to-all patterns that create transient congestion hotspots. These constraints, combined with an operational environment where hardware failures shift from anomalies to frequent occurrences, renders traditionally lightweight operations like initialization and resource management challenging.

Two-wheeler seepage behavior—the aggressive lane-splitting, lateral maneuvering and small-gap acceptance in dense mixed traffic—poses a significant challenge for traffic simulation and autonomous vehicle development in South Asian urban environments. Despite the dominance of two-wheelers in these regions, existing microscopic traffic simulators rely on generic, largely Western-calibrated parameters that fail to reproduce realistic seepage dynamics almost entirely: the default minimum-gap parameter (2.5 m) exceeds the observed mean Indian seepage gap (0.906 m) by 176%, making most real seepage opportunities physically infeasible in simulation. This study proposes a novel real-to-simulation data-driven calibration framework for modeling two-wheeler seepage behavior using 310,110 video frames from the IDD and TIAND real-world traffic datasets. Utilizing a multi-stage quality filtering based on detection confidence, traffic density, and gap realism criteria, we yield 13,219 algorithmically validated seepage events (retained through algorithmic quality scoring across confidence, density, and gap-geometry criteria) with empirical distributions of gap acceptance, lateral positioning, and seepage maneuver types. A percentile-based strategy maps aggressive gap-acceptance thresholds to SUMO simulator parameters, replacing conventional mean-based estimation. Default SUMO produces only 132 seepage events versus 80,553 from the calibrated model—a 610 × increase. The calibration targets seepage emergence at scale and faithful reproduction of the aggressive-tail threshold ( P 10 = 0.272 m) rather than distributional equality across all percentiles. The calibrated mean gap (0.609 m) undershoots the observed mean (0.906 m), a trade-off inherent to parameterized car-following models. On a geo-referenced OpenStreetMap network of Hyderabad the calibrated model generates 49% more seepage events and reproduces the empirical P 10 within 1%. This work provides the first comprehensive, data-driven characterization of two-wheeler seepage behavior and establishes a replicable calibration and validation methodology for realistic mixed-traffic heterogeneous traffic simulation. The framework provides a building block for routing optimization and fleet management where seepage dynamics materially affect travel times and network throughput in different types of traffic environments.

RDMA datacenters are proliferating to meet the demand of emerging workloads such as AI training and inference as well as distributed storage. This trend has opened up a critical knowledge gap: the traffic characteristics of congestion in these networks remain unknown. We do not know, for example, which layers of the network are the most congested, if the network is load balanced effectively, how long congestion events last, and how accurate existing telemetry systems are in capturing congestion. This paper bridges this gap by investigating congestion in a large-scale RDMA datacenter dedicated to distributed AI training. We provide insights into three specific congestion patterns: (a) location and distribution in the network, (b) burstiness, e.g., the duration and synchrony of bursts, and (c) observability using existing telemetry methods. We show, for instance, that the deployment of Priority Flow Control (PFC) in RDMA networks has shifted the location of congestion one level up: from the edge-host in legacy TCP/IP datacenters to the network core in RDMA datacenters. At the same time, we show that the same protocol enables us to observe and understand congestion better, even bursty events. The findings of this research reveal open challenges for measuring, characterizing, and managing congestion in RDMA networks, paving the way for future research.

Modern urban mobility systems are increasingly dependent on precise and scalable simulation platforms to facilitate the design, assessment, and optimization of intelligent transportation systems (ITS). This paper sets forth a thorough benchmarking investigation of three extensively utilized traffic simulation platforms: SUMO (Simulation of Urban MObility), CityFlow, and MATSim, with an emphasis on their computational efficacy within large-scale synthetic traffic scenarios. We conduct an evaluation of these platforms across vehicle volumes ranging from 10 to 10 million vehicles, assessing total runtime, CPU and GPU utilization, and memory consumption on high-performance computing infrastructures. Our results reveal notable architectural trade-offs: SUMO exhibits predictable linear scaling but becomes constrained by CPU limitations at elevated vehicle counts, CityFlow encounters memory limitations beyond 10,000 vehicles, whereas MATSim necessitates meticulous JVM (Java Virtual Machine) optimization to effectively manage large-scale scenarios. This investigation provides essential guidance for researchers and practitioners in the selection of suitable simulation tools for urban-scale traffic modeling and identifies critical computational challenges associated with the scaling of simulations for smart city initiatives.

In datacenters, common incast traffic patterns are challenging because they violate the basic premise of bandwidth stability on which TCP congestion control convergence is built, overwhelming shallow switch buffers and causing packet losses and high latency. To understand why these challenges remain despite decades of research on datacenter congestion control, we conduct an in-depth investigation into high-degree incasts both in production workloads at Meta and in simulation. In addition to characterizing the bursty nature of these incasts and their impacts on the network, our findings demonstrate the shortcomings of widely deployed window-based congestion control techniques used to address incast problems. Furthermore, we find that hosts associated with a specific application or service exhibit similar and predictable incast traffic properties across hours, pointing the way toward solutions that predict and prevent incast bursts, instead of reacting to them.

Managing the performance of thousands of services across millions of servers demands a networking stack that can dynamically adjust protocol settings to match diverse priorities and network characteristics. Moreover, given the constantly evolving nature of services and their requirements, the set of configurable protocols must remain adaptable. However, current host networking stacks lack the necessary flexibility and adaptability. Although eBPF shows promise in this regard, it lacks essential primitives for efficient development and safe deployment of multiple co-existing services.

This work introduces a novel security framework for wireless sensor networks (WSN) based on dynamic duty cycle, which allows nodes to detect their compromised neighbors based on unanticipated fluctuations in network traffic send rate over time. Our framework was assessed by its ability to detect advanced WSN threats (e.g., active, passive, or both attacks). One of the benefits of this framework is that it reduces all threats to unanticipated power dissipation. In other words, the framework assumes any neighbor not conforming to predicted power levels has been communicating with an unauthorized node, and thus is compromised. This threat model is emulated by applying pseudo random but bound (large to small) power dissipations to arbitrary nodes. Simulation results demonstrated that this framework was effective in detecting and isolating compromised sensor nodes.