We present a decade of evolution and production experience with Jupiter datacenter network fabrics. In this period Jupiter has delivered 5x higher speed and capacity, 30% reduction in capex, 41% reduction in power, incremental deployment and technology refresh all while serving live production traffic. A key enabler for these improvements is evolving Jupiter from a Clos to a direct-connect topology among the machine aggregation blocks. Critical architectural changes for this include: A datacenter interconnection layer employing Micro-Electro-Mechanical Systems (MEMS) based Optical Circuit Switches (OCSes) to enable dynamic topology reconfiguration, centralized Software-Defined Networking (SDN) control for traffic engineering, and automated network operations for incremental capacity delivery and topology engineering. We show that the combination of traffic and topology engineering on direct-connect fabrics achieves similar throughput as Clos fabrics for our production traffic patterns. We also optimize for path lengths: 60% of the traffic takes direct path from source to destination aggregation blocks, while the remaining transits one additional block, achieving an average block-level path length of 1.4 in our fleet today. OCS also achieves 3x faster fabric reconfiguration compared to pre-evolution Clos fabrics that used a patch panel based interconnect.

Data Center topologies employ multiple paths among servers to deliver scalable, cost-effective network capacity. The simplest and the most widely deployed approach for load balancing among these paths, Equal Cost Multipath (ECMP), hashes flows among the shortest paths toward a destination. ECMP leverages uniform hashing of balanced flow sizes to achieve fairness and good load balancing in data centers. However, we show that ECMP further assumes a balanced, regular, and fault-free topology, which are invalid assumptions in practice that can lead to substantial performance degradation and, worse, variation in flow bandwidths even for same size flows.

We present the design, implementation, and evaluation of B4, a private WAN connecting Google's data centers across the planet. B4 has a number of unique characteristics: i) massive bandwidth requirements deployed to a modest number of sites, ii) elastic traffic demand that seeks to maximize average bandwidth, and iii) full control over the edge servers and network, which enables rate limiting and demand measurement at the edge.

We present a ”clean-slate” design for a network-layer routing and forwarding system intended to address short-comings of the current Internet Protocol. Our design separates routing from both forwarding and topology discovery; requires only a flat, topology-independent namespace; and allows for policies of both users and service providers to be supported. Channels serve as the primary abstraction, allowing the network topology to be viewed at multiple levels of abstraction using the same identifiers. In this paper we present the basic design, which is based on loose source routing. Our routing and forwarding scheme is part of a larger project to produce a “clean-slate” network layer design.

We study the problem of building an optimal network-layer clustering hierarchy, where the optimality can be defined using three potentially conflicting metrics: state, delay and bandwidth. The problem of network clustering where a node's addresses depends on the node's location (e.g. in the hierarchy) is well studied. We study a problem where network nodes are addressed by specifications that might not be tied to locations in the topology. We propose and compare several distributed clustering algorithms: (i) clustering based solely on topology, (ii) clustering based solely on semantics (node specifications) and (iii) a combination of the above methods (toposemantic network clustering), where we specify a parameter that determines how much the clustering depends on topology and how much on semantics. We show that the toposemantic method yields the best results when we know the right parameter value for a given topology and assignment of specifications. We propose an algorithm that does not require a parameter, but nevertheless yields better results than the first two methods.

We describe a new network service called speccast. Speccast offers a generalized addressing and routing abstraction on which a rich variety of semantic services can be built, and, as such, provides a vehicle for studying the relationships among routing, addressing and topology. Unlike overlay-based systems, we study a more basic problem, in which the topology of the network is given, and there is not necessarily any pre-existing underlying network service. In the speccast model, each packet carries a destination predicate and the network's job is to deliver the packet to all nodes satisfying that predicate. After showing how this generalized routing service subsumes other services both traditional (unicast, multicast) and emerging (publish-subscribe), we present a layered solution for a specific class of simple predicates. We examine the tradeoffs in various forms of our approach, and compare it to existing solutions for unicast and multicast. Studies using transit-stub graphs show that our generic service performs comparably to existing solutions for traditional services, while also effectively supporting new and emerging services.

Abstract Multicast services have been used to transmit multimedia data to large receiver groups. Only recently have counterpart network services been introduced to provide similar scalability and anonymity in the opposite direction (i.e. messages from a group of senders destined for a common receiver). In this paper, we explore how these many‐to‐one services, specifically the concast service, can be used to improve the scalability and performance of multimedia applications. In particular, we show how such services can be used in both the control and data planes to overcome well‐known scalability problems (e.g. with Real Time Protocol) that are difficult to solve effectively with end‐system approaches alone. We validate our solutions by presenting experimental results taken from prototype video and audio applications we designed and implemented. Our initial results show substantial reductions in congestion‐induced loss using these generic services. Copyright © 2003 John Wiley & Sons, Ltd.

Multicast services have been used for many years to transmit multimedia data to large receiver groups. However, only recently have counterpart network services been introduced that provide similar scalability and anonymity in the opposite direction (i.e., messages from a group of senders destined for a common receiver). In this paper, we explore how these emerging services, specifically a concast service (many-to-one communication service), can be used to improve the scalability and performance of multimedia applications. In particular, we show how such services can be used in both the control and data planes to overcome well-known scalability problems (e.g., with RTP) that are difficult to solve effectively with end-system approaches alone. We validate our solutions by presenting experimental results taken from prototype video and audio applications we designed and implemented. Our initial results show a reduction of as much as two orders of magnitude in packet loss rates using these generic services.