Gateways based on the programmable P4 language are becoming a key component in data center traffic management, offering cost-effective solutions for high-throughput environments. However, traditional monitoring techniques like sFlow and NetFlow lack the needed precision to meet the demands of large-scale data centers. In this paper, we presentFlowLog, the first sketch-based and end-to-end flow monitoring system capable of accurate flow size estimation in 400 Gbps production environments. FlowLog integrates the novelByteSketchalgorithm, a transmission subsystem, and a high-speed analysis subsystem, achieving high accuracy even in demanding data center scenarios. Deployed for over six months in ByteDance’s data center with peak bandwidths exceeding 400 Gbps, FlowLog outperforms existing solutions such as Bytehunter sFlow and state-of-the-art sketches in both accuracy and efficiency. Additionally, through real-world deployment, we gained valuable insights that guided improvements in system compatibility, integration ease, and traffic detection. These lessons resulted in a more adaptable system, better handling complex traffic patterns and ensuring minimal overhead during monitoring.

As the number of data center servers grows into the millions and due to the demand for more accurate, rapid and powerful network fault detection and location, the existing Pingmesh-centric monitoring and diagnostic system is not efficient enough. In this paper, we propose ByteTracker, the first agentless probing and diagnostic system for large-scale data center networks. It does not need to deploy probe processes or make any configurations on end hosts, and all probes are launched by a small number of centralized Probers. ByteTracker achieves accurate, real-time probe path tracking with packet mirroring on switches. By reducing end-host probe noise, precisely identifying network timeout probes, accurately tracking probe paths, and marking the failed switch with multiple network timeout probes, ByteTracker can locate network failures with nearly 100% accuracy. We have deployed ByteTracker in all of our data centers for over half a year. During deployment, ByteTracker can detect almost all network anomalies and locate them within 5 seconds with 100% accuracy.

RoCE services are sensitive to failures and bottlenecks, which become more common as the RoCE network scales. To effectively detect and locate these problems independent of service traffic, RoCE networks require a monitoring and diagnostic system based on active probing. However, existing active probing schemes typically rely on a controller to design the probing plan for each server, which is difficult to deploy and has high synchronization overhead in multi-tenant clusters. Fortunately, rail-optimized clusters have become more common in recent years to improve network performance. In these clusters, the controller is unnecessary.

RoCE services are sensitive to network failures and performance bottlenecks, which become more common as the RoCE network scales. In addition, some non-network problems behave like network problems and can waste troubleshooting time. However, existing mechanisms cannot quickly detect and locate network problems or determine whether the service problem is network-related.