The flourishing of Large Language Models (LLMs) calls for increasingly ultra-scale training. In this paper, we share our experience in designing, deploying, and operating our novel Astral datacenter infrastructure, along with operational lessons and evolutionary insights gained from its production use. Astral has three important innovations: (i) a same-rail interconnection network architecture on tier-2, which enables the scaling of LLM training. To physically deploy this high-density infrastructure, we introduce a distributed high-voltage direct current power system and a new air-liquid integrated cooling system. (ii) a full-stack monitoring system featuring cross-host and hierarchical logging correlation, which diagnoses failures at scale and precisely localizes root causes. (iii) an operator-granular forecasting component Seer that efficiently generates operator execution timelines with acceptable accuracy, aiding in fault diagnosis, model tuning, and network architecture upgrading. Astral infrastructure has been gradually deployed over 18 months, supporting LLM training and inference for multiple customers.

An efficient iterative scheduling algorithm for input-queued switches, called round robin with longest queue first (RR/LQF), is proposed in this paper. RR/LQF consists of three phases: report, grant, and accept. In each phase, only a single-bit message per port is sent for reporting a packet arrival, granting an input for packet sending, or accepting a grant. In both the grant and accept phases, scheduling priority is given to the preferred input-output pairs first and the longest virtual output queuing (VOQ) next. The notion of the preferred input-output pair is to keep a global RR schedule among all the inputs and the outputs. By serving the preferred input-output pairs first, the match size tends to be maximized. By serving the longest VOQ next, the match weight is also boosted. When RR/LQF is executed for a single iteration (i.e., RR/LQF-1), we show by simulations that RR/LQF-1 outperforms all the existing single-bit-single-iteration scheduling algorithms. When RR/LQF is executed up to N iterations (i.e., RR/LQF-N), we prove that under any admissible traffic pattern, RR/LQF-N is stable with a speedup of 2-1/N, where N is the switch size. To the best of our knowledge, this is the first work showing that an iterative scheduling algorithm is stable with a speedup less than 2. We then generalize RR/LQF to become a class of algorithms that have the same speedup bound of 2-1/N. Efforts are then made to further reduce the implementation complexity of RR/LQF. To this end, the pipelined RR/LQF and RR/RR, a simpler variant of RR/LQF, are proposed.

A fat-tree network is scalable, but the probability of network faults also increases with the fat-tree size. Faulttolerant routing in fat tree is thus particularly important. Instead of relying on a time-consuming centralized recovery process, a fast local rerouting algorithm is highly desirable. A deflection based fast local rerouting algorithm, which we call it Local Rerouting by Deflection (LRD), was recently proposed to handle the network fault locally and dynamically. However, in LRD, all rerouted packets need to follow an elongated recovery path, which incurs extra propagation delay and additional network congestion. Aiming at addressing the inefficiency of LRD, a new local rerouting algorithm, called Local Rerouting by Backtracking (LRB), is proposed in this paper. With LRB, the faultdetecting switch immediately reroutes the affected packets back to where they came from until a V-turn switch is found. A V-turn switch is a switch from where an alternate minimal path to the destination is found. We show that LRB can minimize packet loss, incur no penalty in path length (except the small amount of backtracked in-flight packets), and treat backtracked packets as implicit fault notification. To ensure efficient implementation of our LRB algorithm, a simple self-routing (instead of table lookup) mechanism is devised to simplify the switch operation. As compared with LRD, we show by simulations that our LRB can minimize the TCP performance degradation caused by network faults.

The feedback-based switch [1] is a load-balanced switch consisting of two stages of crossbar switch fabrics. Each crossbar switch fabric is configured according to a predetermined and periodic seuqence of switch configurations. This eliminates the need for a central scheduler and makes it an excellent candidate for connecting tens of thousands of servers in a data center. But implementing feedback-based switch at the data center scale is challenging. A major issue is that crossbar switch fabrics are not scalable. In this paper, two scalable implementations of the feedback-based switch, dualbanyan network and Clos-banyan network, are proposed. With the dual-banyan implementation, the two crossbar switch fabrics are replaced by two modular banyan networks. To further reduce the dual-banyan network complexity, we merge the last stage of switch modules in the first banyan, the middle-stage ports, and the first stage of switch modules in the second banyan, to form a single stage of shared-memory switch modules. We call the resulting network Clos-banyan. We prove that the feedback-based switch implemented using either dual-banyan or Clos-banyan can realize all the switch configurations required by the original crossbarbased implementation, but at a much lower hardware cost. Since the packet scheduling algorithm is independent of how switch configurations are being realized, different implementations of the feedback-based switch provide the same excellent delaythroughput performance, as that already reported in [1].

An efficient iterative scheduling algorithm for input-queued switches, called Round Robin with Longest Queue First (RR/LQF), is proposed in this paper. RR/LQF only needs a single bit for request, grant and accept messages respectively. The scheduling priority is given to the preferred input/output pairs first. Each single-bit request is actually an indication of a new packet arrival at the specific VOQ. Based on them, each output keeps track of the size of N VOQs destined to it. In the granting phase, if the preferred input's VOQ is empty, an output port j grants the (non-preferred) input that has the longest VOQ (among N VOQs destined to output j). In the accepting phase, if the preferred VOQ is not empty, input port accepts it directly. Otherwise, an input port i accepts the grant received by the long VOQ (among input i). When RR/LQF is executed for a single iteration, we show that RR/LQF outperforms SRR [15] in all simulations conducted. When RR/LQF is executed (up to N iterations) until finding the maximal size matching in input-queued switches, we prove that RR/LQF is stable with a speedup of 2-1/N, where N is the switch size. To the best of our knowledge, this is the first work showing that an iterative scheduling algorithm for input-queued switches can achieve a speedup requirement less than 2. Though the improvement is just 1/N we successfully tighten the speedup bound.

In a data center with TCP/IP communications, it is generally believed that packet-based load balancing is not suitable because the associated packet out-of-order problem will significantly lower the network utilization. In this paper, we first show that if packet-based load balancing is performed properly in a fat-tree based data center, the packet out-of-order problem is not as severe as most researchers believed. This is due to the fact that multiple (minimal) paths between any given pair of servers in a fat-tree are of the same hop count. If packets are evenly routed onto different paths, they will experience similar delay performance. As a result, the packet out-of-order arrivals at the receiver are usually within a small sequence number range. Notably, the fast retransmit (FR) algorithm in TCP will be triggered for resending the “lost” packet if three duplicate ACKs are received (i.e. FR threshold is three). To provide leeway for out-of-order packet arrivals due to packet-based load balancing, we propose to judiciously increase the FR threshold. Simulation results show that FR threshold values between 6 and 9 can effectively suppress unnecessary fast retransmits and at the same time, the impact to real packet losses is minimal. Compared to a flow-based load balancing scheme, we found that our packet-based load balancing with modified TCP consistently provides higher goodput and noticeably smaller delay.

Two major challenges in designing high-speed multicast switches are the expensive multicast switch fabric and the highly complicated central scheduler. While the recent load-balanced switch architecture uses simple unicast switch fabric and does not require a central scheduler, it is only good at handling unicast traffic. In this paper, we extend an existing load-balanced switch called feedback-based two-stage switch to support multicast traffic. In particular, an efficient multicast scheduling algorithm (FTMS) is designed. With FTMS, head-of-line (HOL) packet blocking at each input port is eliminated by adopting “pointer” queues. To cut down queuing delay, packet replication is carried out at middle-stage ports. As compared with other multicast scheduling algorithms, simulation results show that our FTMS always provides the highest throughput.

The feedback-based two-stage switch does not require a central scheduler and can provide close to 100% throughput [3]. But the number of crosspoints required for the two stages of switch fabric is 2N2, and the average packet delay performance (even under light traffic load) is on the order of O(N) slots, where N is the switch size. To improve the performance of feedback-based two-stage switch when N is large, we adopt the Clos network for constructing a large switch from a set of smaller feedback-based switch modules. We call it a Clos-feedback switch. The potential problem of packet mis-sequencing is solved by using application-flow based load balancing. With recursive decomposition, a Clos network can degenerate into a Benes network. We show that for a Clos-feedback switch, the number of crosspoints required is reduced to 4N(2 log2N-1) and the average packet delay is cut down to O(log2N) slots.

A general approach of designing input-queued multicast switch is to employ multicast switch fabric, where packets can be replicated inside the switch fabric. As compared with unicast switch fabric, the achievable traffic rate region of a switch can be increased, but it is still less than the admissible traffic rate region. In other words, achieving 100% throughput for any admissible multicast traffic pattern is not possible. In this paper, we first revisit the fundamental problems faced by input-queued switch in supporting multicast traffic. We then argue that multicast switch fabric is not necessary if a load-balanced approach is followed. Accordingly, an existing load-balanced two-stage switch architecture [12], consisting of unicast switch fabrics, can be adopted to provide 100% throughput for any admissible multicast traffic pattern. Since the two-stage switch requires no speedup in both switch fabric and packet buffers, we consider it a two-stage input-queued switch. It can be seen that its implementation complexity is much lower than conventional (single-stage) input-queued multicast switches. As compared with the work in [12], our approach is more systematic and we propose a more effective load balancing mechanism.

Due to the massive use of parallel and distributed operations of inputs and outputs, iterative scheduling algorithms are attractive in finding a maximal size matching for an input-queued switch. For constructing a large high-speed switch, a distributed multi-chip implementation of an iterative scheduling algorithm should be followed. Since different chips may locate on different switch linecards and linecards can be separated by tens of meters, the propagation delay between chips/linecards is non-negligible. This calls for a pipelined implementation of a single-iteration scheduling algorithm. In this paper, an efficient, pipelined single-iteration algorithm called Distributed Longest Queue First (D-LQF) is proposed. In D-LQF, exhaustive service policy is adopted for reusing the matched input-output pairs in the previous time slot. To avoid incorrectly granting an empty VOQ from transmission (caused by inter-chip latency), each output keeps track of the lengths of all VOQs destined to it. As compared with other single-iteration scheduling algorithms, extensive simulation results show that D-LQF provides the best delay-throughput performance.