Parameter Server (PS) and Ring-AllReduce (RAR) are two widely utilized synchronization architectures in multiworker Deep Learning (DL), also referred to as Distributed Deep Learning (DDL). However, PS encounters challenges with the “incast” issue, while RAR struggles with problems caused by the long dependency chain. The emerging In-network Aggregation (INA) has been proposed to integrate with PS to mitigate its incast issue. However, such PS-based INA has poor incremental deployment abilities as it requires replacing all the switches to show significant performance improvement, which is not costeffective. In this study, we present the incorporation of INA capabilities into RAR, called RAR with In-Network Aggregation (Rina), to tackle both the problems above. Rina features its agent-worker mechanism. When an INA-capable ToR switch is deployed, all workers in this rack run as one abstracted worker with the help of the agent, resulting in both excellent incremental deployment capabilities and better throughput. We conducted extensive testbed and simulation evaluations to substantiate the throughput advantages of Rina over existing DDL training synchronization structures. Compared with the state-of-the-art PS-based INA methods ATP, Rina can achieve more than$\mathbf{5 0 \%}$throughput with the same hardware cost.

RDMA over Converged Ethernet (RoCE) has been widely used in datacenter networks and it relies on Priority Flow Control (PFC) to ensure a lossless network. However, PFC brings certain side effects, such as Head-of-Line (HoL) blocking, congestion spreading, and deadlock. Existing solutions demonstrate inherent limitations: either fail to completely eliminate the adverse impacts of PFC or introduce extra challenges. In light of these observations, this paper proposes a novel and practical scheme, Restricted Priority Flow Control (R-PFC). R-PFC consists of two parts: one-hop PFC and Virtual Next Output Queue (VNOQ). Instead of passively regarding PFC as a tool to guarantee a lossless network, one-hop PFC proactively employs PFC in a restrictive manner to minimize packet loss while limiting the spread of congestion within one hop. To further enhance the one-hop PFC, the fine-grained VNOQ solves the HoL blocking issue. We theoretically prove that R-PFC does not lead to deadlock and evaluate the performance of R-PFC under typical datacenter network scenarios in ns3 simulations. The results show that R-PFC outperforms both lossless and lossy networks by 43.76% and 39.46% on average.

RoCEv2 (RDMA over Converged Ethernet version 2) is typically used in a PFC-enabled lossless network for high performance, but PFC can cause side effects, such as head-of-line (HoL) blocking and congestion spreading. Optimizing packet loss recovery mechanisms in lossy networks can enhance RDMA network performance, but packet loss can increase FCT of short flows and waste network resources. This paper proposes a novel concept of semi-lossless networks to exploit the advantages of lossless and lossy networks, and reduce their negative effects. Our proposed solution, Selective PFC (S-PFC), implements semi-lossless networks in two dimensions. First, S-PFC ensures no packet loss for short flows while timely dropping long flows. Second, S-PFC guarantees no packet loss at the network edge to prevent premature packet loss and unnecessary resource waste. Typical data center network scenarios and large-scale simulations show that S-PFC can accommodate different traffic demands effectively.