We describe our experiences using formal methods and automated theorem proving for network operation at scale. The experiences are based on developing and applying the SecGuru and RCDC (Reality Checker for Data Centers) tools in Azure. SecGuru has been used since 2013 and thus, is arguably a pioneering industrial deployment of network verification. SecGuru is used for validating ACLs and more recently RCDC checks forwarding tables at Azure scale. A central technical angle is that we use local contracts and local checks, that can be performed at scale in parallel, and without maintaining global snapshots, to validate global properties of datacenter networks. Specifications leverage declarative encodings of configurations and automated theorem proving for validation. We describe how intent is automatically derived from network architectures and verification is incorporated as prechecks for making changes, live monitoring, and for evolving legacy policies. We document how network verification, grounded in architectural constraints, can be integral to operating a reliable cloud at scale.

Models of Internet routing are critical for studies of Internet security, reliability and evolution, which often rely on simulations of the Internet's routing system. Accurate models are difficult to build and suffer from a dearth of ground truth data, as ISPs often treat their connectivity and routing policies as trade secrets. In this environment, researchers rely on a number of simplifying assumptions and models proposed over a decade ago, which are widely criticized for their inability to capture routing policies employed in practice.

A fundamental problem in 802.11 wireless networks is to accurately determine the cause of packet losses. This becomes increasingly important as wireless data rates scale to Gbps, where lack of loss differentiation leads to higher loss in throughput. Recent and upcoming high-speed WLAN standards, such as 802.11n and 802.11ac, use frame aggregation and block acknowledgements for achieving efficient communication. This paper presents BLMon, a framework for loss differentiation, that uses loss patterns within aggregate frames and aggregate frame retries to achieve accurate and low overhead loss differentiation. Towards this end, we carry out a detailed measurement study on a real testbed to ascertain the differences in loss patterns due to noise, collisions, and hidden nodes. We then devise metrics to quantitatively capture these differences. Finally, we design BLMon, which collectively uses these metrics to infer the cause of loss without requiring any out-of-band communication, protocol changes, or customized hardware support. BLMon can be readily deployed on commodity devices using only driver-level changes at the sender-side. We implement BLMon in the ath9k driver and using real testbed experiments, show that it can provide up to 5× improvement in throughput.