Rail-optimized network fabrics have become the de facto data-center scale-out fabric for large-scale ML training. However, the use of high-radix electrical switches to provide all-to-all connectivity in rails imposes substantial power and cost. We propose a rethinking of the rail abstraction by retaining its communication semantics, but realizing it using optical circuit switches. The key challenge is that optical switches support one-to-one connectivity at a time, limiting the fan-out of traffic in ML workloads using hybrid parallelisms. We overcome this through parallelism-driven rail reconfiguration, which exploits the non-overlapping communication phases of different parallelism dimensions. This time-multiplexes a single set of physical ports across circuit configurations tailored to each phase within a training iteration. We design and implement Opus, a control plane that orchestrates this in-job reconfiguration of photonic rails at parallelism phase boundaries, and evaluate it on a physical OCS testbed, the Perlmutter supercomputer, and in simulation at up to 2,048 GPUs. Our results show that photonic rails can achieve over 23× network power reduction and 4× cost savings while incurring only modest training overhead at production-relevant OCS reconfiguration latencies.

This paper demonstrates a new dimension in packet programming and processing by leveraging New IP technology since applications are sensitive to different types of network constraints. For instance, emerging industry operations, manufacturing, and autonomics are limited by the stochastic quality of services and inflexible address structures. Instead, they require efficiency and deterministic networks. In this paper, we propose a programmable data plane for New IP packet processing and show how network headers could evolve in the future. We demonstrate the implementation of New IP stack to encompass three goals: (1) address customization: applications and routers can forward packets between hosts with different address formats. (2) design an end-to-end model to meet service delivery guarantees: routers implement various in-network New IP contracts as described by the applications. (3) Rapid experimentation of the New IP components. With New IP, developers can describe packet processing functionalities without depending on the specifics of the underlying hardware. Our implementation of New IP stack uses the existing toolsets and capabilities of the Linux platform, such as eXpress Data Path (XDP) and Traffic Control (TC) subsystem. It consists of an end-to-end solution with a new network stack on the host side and a corresponding packet processing and forwarding engine on the network. It is validated using Network Stack Tester (NeST), a lightweight tool built on Linux network namespaces.

The design of a transport protocol for high precision industrial networks is a complex problem since the industrial applications are resource-critical, time-critical, session-less, and safety-critical. These requirements make existing end-to-end transport mechanisms unsuitable for use in the industry verticals that require control-systems type communication. This paper proposes a lightweight, connection-less, and reliable protocol called TinTin. It utilizes the in-network capabilities to compensate for end device constraints. The mechanisms adopted in TinTin aim to reduce the communication overheads associated with well-known connection-oriented protocols. This paper discusses the design considerations of TinTin in detail and presents an early stage proof-of-concept implementation of TinTin with New IP.