The rapid explosion of Artificial Intelligence (AI) workloads utilizing a growing number of accelerators has placed unprecedented demand on the network. These workloads typically leverage Remote Direct Memory Access (RDMA) and require a high-performance network fabric. While many purpose-built cloud networking solutions can provide high performance, efficiently utilizing these costly infrastructures require a fabric that is multi-tenant for ease of consumption. Furthermore, the fabric must be resilient to faults and for operational manageability. Resilient cloud networks typically employ mature Ethernet segmentation techniques over Clos topologies with Equal Cost Multi Path (ECMP) routing. ECMP hashes flows to paths, which in case of collisions can significantly degrade performance for large RDMA over Converged Ethernet (RoCE) flows. To mitigate ECMP penalties, we evaluate routing strategies with varying levels of operational complexity. We explore load balancing and path pinning solutions that leverage non-proprietary, mature technologies over commodity Ethernet. Our evaluation follows a three-fold strategy, focusing on the key dimensions of performance, resiliency, and operational complexity. By applying this methodology to representative implementations, we highlight the trade-offs. While all techniques are resilient, path pinning-based solutions excel at performance but introduce greater complexity. Specifically, path pinning achieves up to 1.6× improvement over ECMP for RoCE test traffic and up to 2.5× for NCCL AllReduce. These results validate the promising performance benefits of path pinning and highlight the need to explore less complex implementations for broader adoption. Our methodology can be used to rigorously evaluate future implementations in support of AI network design.

Vela is a cloud-native system designed for LLM training workloads built using off-the-shelf hardware, Linux KVM-based virtualization, and a virtualized RDMA over Converged Ethernet (RoCE) network. Vela virtual machines (VMs) support peer-to-peer DMA between the GPUs and SRIOV-based network interface. In this paper, we share Vela's key architectural aspects with details from an NVIDIA A100 GPU-based deployment in one of the IBM Cloud data centers. Throughout the paper, we share insights and experiences from designing, building, and operating the system over a ~2.5 year timeframe to highlight the capabilities of readily available software and hardware technologies and the improvement opportunities for future AI systems, thereby making AI infrastructure more accessible to a broader community. As we evaluated the system for performance at ~1500 GPU scale, we achieved ~80% of the ideal throughput while training a 50 billion parameter decoder model using model parallelism, and ~70% per GPU FLOPS compared to a single VM with the High-Performance Linpack benchmark.

Modern Artificial Intelligence (AI) and HighPerformance Computing (HPC) workloads impose diverse demands on data center networks in terms of both performance and reliability. Applications such as inferencing are usually compute-bound, while distributed training and HPC workloads are network-bound, requiring high bandwidth and low, predictable latency. In terms of reliability, some workloads require seamless failure recovery, while others can tolerate failures through checkpointing or application/transport-layer reliability. In this paper, we address the challenge of designing a highperformance network for private data centers at scales of 1-2K Graphics Processing Units (GPUs) or other accelerators, supporting a range of workloads over Remote Direct Memory Access (RDMA), RDMA over Converged Ethernet (RoCE), and Transmission Control Protocol (TCP) with off-the-shelf hardware and software. We first present a network control plane design that accommodates flexible reliability and performance requirements, and we present our approach across key system components to efficiently utilize multiple equal-cost paths in a two-level leafspine topology. We then use simulations to guide our design choices, including a 1:4 speed ratio between server-facing and spine-facing ports, along with switch partitioning at the spine layer to mitigate the impact of flow collisions. Leveraging Ansible automation for efficient configuration management, we integrate these findings into a 12 -node cluster with 8 GPUs and 1.6 Tbps bandwidth per server. Using network micro-benchmarks that mimic the demands of intensive workloads, our results show that the network sustains near-line-rate throughput under traffic patterns where up to two-thirds of traffic traverses the spine, while also supporting a flexible reliability model.

Cloud networks support workloads with diverse characteristics. A key challenge facing cloud providers is how to meet the stringent performance and security needs of these diverse applications running over a shared infrastructure. To address this challenge, some providers build for peak capacity or even build dedicated clusters with specialized networks, which may be under-utilized at times. We propose a virtualization approach that customizes data center network resources to the needs of applications. Our approach is based on slicing data center network resources on-demand and customizing these slices to target workloads. Such slices can grow or shrink dynamically and programmatically based on workload demands. This elasticity provides a more efficient solution over building dedicated clusters. In our approach, a slice can be customized to a given set of workloads with similar security and performance requirements carved out of the underlying network. It leverages a software-defined underlay network controller and segment routing for fine-grained path control and service chaining. We have implemented a prototype of our fabric virtualization solution based on network slicing. In this paper, we first present the architecture of our prototype. Second, we present empirical results of slice provisioning times in networks of varying sizes and switch operating systems. Empirical results indicate that our prototype can support slice provisioning in the order of tens to hundreds of seconds and can meet the provisioning requirements of production networks.

Data centre applications for batch processing (e.g. map/reduce frameworks) and online services (e.g. search engines) scale by distributing data and computation across many servers. They typically follow a partition/aggregation pattern: tasks are first partitioned across servers that process data locally, and then those partial results are aggregated. This data aggregation step, however, shifts the performance bottleneck to the network, which typically struggles to support many-to-few, high-bandwidth traffic between servers.

The original Internet architecture lacked the concept of a flow, and considered each traffic as a set of packets. In this short paper, we rethink this concept inside middlebox-based platform and handle each traffic as a whole block instead of packets. We design a whole system where each input packet matching some criteria is placed in a specific structure which is shared between all processing modules that interact in a parallel manner with this flow. Thus, this new design improves flexibility of traffic and also increases the flow processing performances.