Advancements in hardware architecture and system design have enabled the transformational pivot towards cloud-computing. The rising cost of on-premise, vertical scaling, and maintenance combined with the rise of workload heterogeneity have fueled this paradigm shift. Furthermore, increasingly demanding use-cases for storage and High-Performance Computing (HPC) as well as the emergence of new workloads such as machine learning and big-data have motivated research into network traffic analysis, resource disaggregation, job scheduling, and orchestration in a bid to reduce total cost while maintaining high performance. A prime focus of this research is the communication performance of collectives, which comprise a significant portion of the communication of many of the aforementioned workloads. In this paper, we characterize the performance of MPI_Allreduce, which is used extensively in HPC and Deep-Learning (DL) training workloads in cloud environments. We demonstrate several key insights including that the Ring algorithm performs better than the Rabenseifner algorithm under conditions of high congestion and packet drops. We further illustrate detailed performance analysis under different assumptions and provide recommendations on how to address issues that can manifest.

The explosive growth in cloud-computing and overall data center system growth has created an unprecedented demand on system architects and designers to continuously develop more complex system networks to effectively satisfy the insatiable appetite to process, move, and store large amounts of data. Nonlinear system behavior caused by emerging workloads and use-cases, varying end-to-end congestion protocols, and heterogeneity in the various compute and storage capabilities of custom designed accelerators further compounds the design problem. Modern simulation methodologies lack a cohesive and efficient framework to address the interoperability of the intersecting layers at scale. In this paper, we present a simulation framework for evaluating congestion control protocols. Furthermore, we present a set of optimizations that enable analysis for longer simulated times and at network scales up to 128K nodes, which is vital for proper analysis of workloads that require long run times (e.g., AI training) or workloads that are known to have scaling issues (e.g., RDMA). Specifically, we evaluate congestion control performance at various scales, study the implications of topology scaling on congestion, and the performance impact of simultaneous heterogeneous protocols.

This paper proposes and evaluates Sharing/Timing Adaptive Push (STAP), a dynamic scheme for preemptively sending data from producers to consumers to minimize critical-path communication latency. STAP uses small hardware buffers to dynamically detect sharing patterns and timing requirements. The scheme applies to both intra-node and inter-socket directory-based shared memory networks.

This paper shows that using SRIOV for InfiniBand can enable virtualized HPC, but only if the NIC tunable parameters are set appropriately. In particular, contrary to common belief, our results show that the default policy of aggressive use of interrupt moderation can have a negative impact on the performance of InfiniBand platforms virtualized using SR-IOV. Careful tuning of interrupt moderation benefits both Native and VM platforms and helps to bridge the gap between native and virtualized performance. For some workloads, the performance gap is reduced by 15-30%.