Interconnection network in HPC is becoming a bottleneck due to increasing traffic load. We model adaptive routing mechanisms and prove that even with advanced adaptive routing, static networks like Dragonfly cannot handle non-uniform traffic efficiently, let alone the frequently changing non-uniform traffic. Therefore, it requires architectural changes for network-wide improvements, e.g., reconfigurable networks.Existing reconfigurable networks hardly support agile reaction to traffic changes with little impact on network. Therefore, we propose MUSE1, a Dragonfly-based runtime incrementally reconfigurable network to enable a small number of link adjustments for agility and little impact on transmitting flows during every reconfiguration with optical circuit switch (OCS).Simulations with both synthetic traffic and real-world workloads prove that MUSE can prevent saturation under typical traffic patterns that cause congestion in static Dragonfly. MUSE is 30-55% better than static Dragonfly and Flexfly w.r.t commonly used performance metrics like flow completion time (FCT). We also build a MUSE prototype and demonstrate that MUSE enables 20-30% less application finish time (AFT).

RDMA over Converged Ethernet (RoCE) has been widely used in datacenter networks and it relies on Priority Flow Control (PFC) to ensure a lossless network. However, PFC brings certain side effects, such as Head-of-Line (HoL) blocking, congestion spreading, and deadlock. Existing solutions demonstrate inherent limitations: either fail to completely eliminate the adverse impacts of PFC or introduce extra challenges. In light of these observations, this paper proposes a novel and practical scheme, Restricted Priority Flow Control (R-PFC). R-PFC consists of two parts: one-hop PFC and Virtual Next Output Queue (VNOQ). Instead of passively regarding PFC as a tool to guarantee a lossless network, one-hop PFC proactively employs PFC in a restrictive manner to minimize packet loss while limiting the spread of congestion within one hop. To further enhance the one-hop PFC, the fine-grained VNOQ solves the HoL blocking issue. We theoretically prove that R-PFC does not lead to deadlock and evaluate the performance of R-PFC under typical datacenter network scenarios in ns3 simulations. The results show that R-PFC outperforms both lossless and lossy networks by 43.76% and 39.46% on average.

Distributed Machine Learning (DML) systems are utilized to enhance the speed of model training in data centers (DCs) and edge nodes. The Parameter Server (PS) communication architecture is commonly employed, but it faces severe long-tail latency caused by many-to-one “incast” traffic patterns, negatively impacting training throughput. To address this challenge, we design the Loss-tolerant Transmission Protocol (LTP), which permits partial loss of gradients during synchronization to avoid unneeded retransmission and contributes to faster synchronization per iteration. LTP implements loss-tolerant transmission through out-of-order transmission and out-of-order Acknowledges (ACKs). LTP employs Early Close to adjust the loss-tolerant threshold based on network conditions and bubble-filling for data correction to maintain training accuracy. LTP is implemented by C++ and integrated into PyTorch. Evaluations on a testbed of 8 worker nodes and one PS node demonstrate that LTP can significantly improve DML training task throughput by up to 30x compared to traditional TCP congestion controls, with no sacrifice to final accuracy.