Silent packet drops and packet corruptions, which are caused by faulty network elements and hurt performances of cloud applications, are common in data centers but hard to detect and localize. Existing solutions based on active probes introduce additional probe traffic and are constrained by probe rate, while solutions based on passive traffic monitoring measure the entire network traffic, and are generally unable to pinpoint the locations where packet losses happen. In this paper, we presentCanary, a system for detecting and localizing network faults with partial traffic monitoring. Canary employs a lightweight and adaptive mechanism to detect packet losses by monitoring a small set of large-sized network flows, and it ensures that on each network path, a sufficient number of packets are monitored by upstream and downstream switches. In addition, Canary encodes information of the path that a packet travels along within its header, and by leveraging path information of the lost packets, Canary is capable to localize network faults with high accuracy. We theoretically prove the effectiveness of our proposed method, and prototype Canary with P4 on commodity hardware programmable switches. Results from extensive experiments driven by real-world traffic show that Canary is lightweight regarding measurement overhead, robust under traffic dynamics, and is accurate in detecting and localizing faulty network links. In particular, comparing with the state-of-the-art solutions, Canary reduces the memory overhead by over 97% under$10^{-2}$link loss rate, and increases the F1-score in localizing the faulty links by over 20% on a$k=8$fat-tree data center network.

Abstract A sketch-based method is promising for traffic monitoring in data center networks. Existing data plane programming model (e.g. P4) assumes target switch as one single pipeline, while state-of-the-art programmable switches actually contain multiple independent pipelines. The status quo approach for deploying a sketch-based measurement application on a multi-pipeline switch is to deploy a sketch instance in each pipeline individually. However, under multi-path routing, such a naive approach leads to poor accuracy. To overcome this problem, in this paper, we present Confluence, a sketch-based network measurement system for multi-pipeline switches. For monitoring network flows that have packets arrived in bursts and spread over multiple pipelines, Confluence introduces novel data structures to collect short-term traffic statistics in ingress pipelines, and converge the measurement data to egress pipelines. Confluence is carefully designed under the switch hardware constraints, and in particular, to resolve the circular dependency in querying and updating a flow’s measurement data from sketch buckets, we propose a novel algorithm and theoretically prove its effectiveness. Both theoretical analysis and experiments driven by real-world traffic traces show that Confluence delivers higher measurement accuracies than existing solutions, especially in the critical task of detecting heavy hitters. Assessment on hardware switch suggests that Confluence is practical for real-world deployment.

Sketch-based method has emerged as a promising direction for per-flow measurement in data center networks. Usually in such a measurement system, a sketch data structure is placed as a whole at one switch for counting all passing packets, but when summarizing measurement results from multiple switches, the overall accuracy is generally constrained by a few individual switches with small-sized sketches due to their limited memory resources. To address this problem, in this paper, we present Distributed Sketch, a new method for per-flow network measurement in data center networks. In Distributed Sketch, each network path is associated with a logical sketch, whose data structure is collectively maintained by all the switches along the path; meanwhile, each switch multiplexes its physical sketch to the constructions of the logical sketches of all the paths it belongs to. With Distributed Sketch, switches collaborate to measure network flows, and the network-wide measurement workload is fairly distributed among all the switches in the network. We implement Distributed Sketch with P4 on commodity hardware programmable switch, and in particular, to overcome the limitation that hardware switches do not support float-point computation, we present an optimal approximation method that involves only integer operations. We also propose an In-band Network Telemetry (INT) based method for addressing the challenges in deploying Distributed Sketch in large-scale data centers. Experiment results and theoretical analysis show that our proposed method is lightweight regarding measurement overhead, and by aggregating and making fair uses of resources from all the switches in the network, Distributed Sketch achieves a higher measurement accuracy compared with the state-of-the-art solutions.