We describe our experience with Fathom, a system for identifying the network performance bottlenecks of any service running in the Google fleet. Fathom passively samples RPCs, the principal unit of work for services. It segments the overall latency into host and network components with kernel and RPC stack instrumentation. It records these detailed latency metrics, along with detailed transport connection state, for every sampled RPC. This lets us determine if the completion is constrained by the client, network or server. To scale while enabling analysis, we also aggregate samples into distributions that retain multi-dimensional breakdowns. This provides us with a macroscopic view of individual services. Fathom runs globally in our datacenters for all production traffic, where it monitors billions of TCP connections 24x7. For five years Fathom has been our primary tool for troubleshooting service network issues and assessing network infrastructure changes. We present case studies to show how it has helped us improve our production services.

Despite significant advances, supporting high-quality large video conferences at a low cost remains a significant challenge due to stringent performance requirements, limited and heterogeneous client resources, and dynamic traffic demands. In this paper, we develop a simple yet effective valiant multicast routing to select application-layer routes and adapt streaming rates according to the current network condition. It consists of four novel components: (i) a valiant multicast routing using two random choices to effectively balance the load in the presence of uncertainty about the clients' load, (ii) a scheme to cluster clients based on their delay and adapt valiant multicast routing based on both upload capacity and locality, (iii) an approach to further leverage resources from other peers or nodes in content distribution network (CDN) to enhance performance, and (iv) a simple distributed scheme to adapt streaming rates according to the current network resources. Our real implementation and experiments show that our approach significantly out-performs existing multicast routing schemes and quickly adapts to changing traffic demands and network conditions.

This paper proposes and studies a system, called CloudFlex, which transparently taps cloud resources to serve application requests that exceed capacity of internal infrastructure. CloudFlex operates as a feedback control system with two key interacting components: load balancer and controller. We focus on operational optimality and stability of the system, highlight the tradeoffs between cost and responsiveness, and address important design considerations such as choke point detection that are critical in avoiding pathological system operations. For evaluation, we develop a prototype of CloudFlex on our testbed comprising servers of our enterprise data center and Amazon EC2 instances.

We present VCD, a novel system for enabling high-bandwidth content distribution in vehicular networks. In VCD, a vehicle opportunistically communicates with nearby access points (APs) to download the content of interest. To fully take advantage of such transient contact with APs, we proactively push content to the APs that the vehicles will likely visit in the near future. In this way, vehicles can enjoy the full wireless capacity instead of being bottle-necked by the Internet connectivity, which is either slow or even unavailable. We develop a new algorithm for predicting the APs that will soon be visited by the vehicles. We then develop a replication scheme that leverages the synergy among (i) Internet connectivity (which is persistent but has limited coverage and low bandwidth), (ii) local wireless connectivity (which has high bandwidth but transient duration), (iii) vehicular relay connectivity (which has high bandwidth but high delay), and (iv) mesh connectivity among APs (which has high bandwidth but low coverage). We demonstrate the effectiveness of VCD system using trace-driven simulation and Emulab emulation based on real taxi traces. We further deploy VCD in two vehicular networks: one using 802.11b and the other using 802.11n, to demonstrate its effectiveness.