This paper presents the first hardware implementation of bittide, a decentralized clock synchronization mechanism for achieving logical synchrony in distributed systems. We de-tail the design and implementation of an 8-node bittide network using off-the-shelf FPGA boards and adjustable clock sources. Through experiments with various network topologies, including fully connected, hourglass, and cube, we demonstrate the ef-fectiveness of bittide in aligning node frequencies and bounding buffer excursions. We collect and analyze frequency, buffer oc-cupancy, and logical latency data, validating the hardware's per-formance against theoretical predictions and simulations. Our results show that bittide achieves tight frequency alignment, ro-bustly handles varying physical latencies, and establishes a con-sistent notion of logical time across the network, enabling pre-dictable distributed computation at scale with zero in-band over-head.

We introduce logical synchrony, a framework that allows distributed computing to be coordinated as tightly as in synchronous systems without the distribution of a global clock or any reference to universal time. We develop a model of events called a logical synchrony network, in which nodes correspond to processors and every node has an associated local clock which generates the events. We construct a measure of logical latency and develop its properties. A further model, called a multiclock network, is then analyzed and shown to be a refinement of the logical synchrony network. We present the bittide mechanism as an instantiation of multiclock networks, and discuss the clock control mechanism that ensures that buffers do not overflow or underflow. Finally we give conditions under which a logical synchrony network has an equivalent synchronous realization.

We discuss distributed reframing control of bittide systems. In a bittide system, multiple processors synchronize by monitoring communication over the network. Processors remain in logical synchrony by controlling the timing of frame transmissions. The protocol for doing this relies upon an underlying dynamic control system where each node makes only local observations and performs no direct coordination with other nodes. In this paper we develop a control algorithm based on the idea of buffer centering, which allows all nodes to maintain small buffer offsets while also requiring very little state information. We demonstrate that with buffer centering we can achieve separate control of frequency and phase, allowing frequencies to be syntonized and also buffers to maintain desired offsets rather than combining their control via a proportional-integral controller. The minimalism of this approach offers the potential to simplify both boot processes and failure handling.

This paper describes our effort to build extensible routers using a combination of general-purpose and network processors. We emphasize five overriding challenges that dictate our design decisions: (1) optimal resource allocation; (2) efficient but flexible scheduling of the CPU; (3) maintaining overall router robustness; (4) maximizing router performance; and (5) providing sufficient extensibility to enable the injection of new functionality into the router. We adopt a hierarchical architecture, in which packet flows traverse a range of processing/forwarding paths, thereby partitioning hardware and software in concert. This paper both presents the architecture, and describes our experiences implementing the architecture and addressing the five design challenges in a prototype built from Intel IXP 1200 and a Pentium. Copyright © 2005 John Wiley & Sons, Ltd.

Recent efforts to add new services to the Internet have increased interest in software-based routers that are easy to extend and evolve. This paper describes our experiences using emerging network processors---in particular, the Intel IXP1200---to implement a router. We show it is possible to combine an IXP1200 development board and a PC to build an inexpensive router that forwards minimum-sized packets at a rate of 3.47Mpps. This is nearly an order of magnitude faster than existing pure PC-based routers, and sufficient to support 1.77Gbps of aggregate link bandwidth. At lesser aggregate line speeds, our design also allows the excess resources available on the IXP1200 to be used robustly for extra packet processing. For example, with 8 × 100Mbps links, 240 register operations and 96 bytes of state storage are available for each 64-byte packet. Using a hierarchical architecture we can guarantee line-speed forwarding rates for simple packets with the IXP1200, and still have extra capacity to process exceptional packets with the Pentium. Up to 310Kpps of the traffic can be routed through the Pentium to receive 1510 cycles of extra per-packet processing.

Swarm is a storage system that provides scalable, reliable, and cost-effective data storage. Swarm is based on storage servers, rather than file servers; the storage servers are optimized for cost-performance and aggregated to provide high-performance data access. Swarm uses a striped log abstraction to store data on the storage servers. This abstraction simplifies storage allocation, improves file access performance, balances server loads, provides fault-tolerance through computed redundancy, and simplifies crash recovery. We have developed a Swarm prototype using a cluster of Linux-based personal computers as the storage servers and clients; the clients access the servers via the Swarm-based Sting file system. Our performance measurements show that a single Swarm client can write to two storage servers at 3.0 MB/s, while four clients can write to eight servers at 16.0 MB/s.