We describe our experience with Fathom, a system for identifying the network performance bottlenecks of any service running in the Google fleet. Fathom passively samples RPCs, the principal unit of work for services. It segments the overall latency into host and network components with kernel and RPC stack instrumentation. It records these detailed latency metrics, along with detailed transport connection state, for every sampled RPC. This lets us determine if the completion is constrained by the client, network or server. To scale while enabling analysis, we also aggregate samples into distributions that retain multi-dimensional breakdowns. This provides us with a macroscopic view of individual services. Fathom runs globally in our datacenters for all production traffic, where it monitors billions of TCP connections 24x7. For five years Fathom has been our primary tool for troubleshooting service network issues and assessing network infrastructure changes. We present case studies to show how it has helped us improve our production services.

This paper describes the design and implementation of a new operating system authorization architecture to support trustworthy computing. Called logical attestation, this architecture provides a sound framework for reasoning about run time behavior of applications. Logical attestation is based on attributable, unforgeable statements about program properties, expressed in a logic. These statements are suitable for mechanical processing, proof construction, and verification; they can serve as credentials, support authorization based on expressive authorization policies, and enable remote principals to trust software components without restricting the local user's choice of binary implementations.

Streamline is a stream-based OS communication subsystem that spans from peripheral hardware to userspace processes. It improves performance of I/O-bound applications (such as webservers and streaming media applications) by constructing tailor-made I/O paths through the operating system for each application at runtime. Path optimization removes unnecessary copying, context switching and cache replacement and integrates specialized hardware. Streamline automates optimization and only presents users a clear, concise job control language based on Unix pipelines. For backward compatibility Streamline also presents well known files, pipes and sockets abstractions. Observed throughput improvement over Linux 2.6.24 for networking applications is up to 30-fold, but two-fold is more typical.

Beltway buffers are operating system I/O paths optimised for high-throughput network applications. The key architectural feature of Beltway buffers is that all I/O takes place in long-lived, allocation-free, shared ringbuffers. Advantages of this design are (1) improved throughput through system-wide copy, context-switch and allocation avoidance and judicious use of the data cache, (2) transparent integration of peripheral hardware and (3) simplicity and familiarity due to comprehensive use of the POSIX file interface for accessing streams.

Multi-gigabit speeds are reaching the periphery of the network, where both hard- and software are unprepared for these high rates. With wider pipes available, network uses are also diversifying: time-constrained streaming media, low-latency telephony and high-throughput bulk transfers have to contend for the same overloaded resources on the host. To top it off, a heightened awareness of the need for security is making host-side firewalling and intrusion detection commonplace.

Self-management has often been proposed as a means to reduce the growing complexity of administration in distributed systems. We argue that this can be achieved through aggressive automation of management tasks. To reach a high level of automation, we propose to take an inference-based approach: codify best practices so that they can be reasoned about and adapted at runtime. Concerns specific to distributed systems are dealt with by the innate support for knowledge sharing. We introduce the methodology along with a reference architecture. The method's validity is tested by applying a preliminary implementation to a handful of practical problems.

We describe a drop-in replacement for net-snmp, known as 'Splash' (SNMP Plus a Lightweight API for SNAP Handling), which adds support for efficient mobile agents to the standard SNMP agent. Experiments show that the SNAP (Safe and Nimble Active Packets) mobile agent engine is comparable in speed to SNMP, even for the simple polling scenarios for which SNMP is optimized, yet also provides flexibility that allows it to outperform SNMP in more complex scenarios. Splash allows network operators to select the monitoring paradigm (polling with SNMP, or a mobile agent approach) most appropriate for a given situation.