The rapid explosion of Artificial Intelligence (AI) workloads utilizing a growing number of accelerators has placed unprecedented demand on the network. These workloads typically leverage Remote Direct Memory Access (RDMA) and require a high-performance network fabric. While many purpose-built cloud networking solutions can provide high performance, efficiently utilizing these costly infrastructures require a fabric that is multi-tenant for ease of consumption. Furthermore, the fabric must be resilient to faults and for operational manageability. Resilient cloud networks typically employ mature Ethernet segmentation techniques over Clos topologies with Equal Cost Multi Path (ECMP) routing. ECMP hashes flows to paths, which in case of collisions can significantly degrade performance for large RDMA over Converged Ethernet (RoCE) flows. To mitigate ECMP penalties, we evaluate routing strategies with varying levels of operational complexity. We explore load balancing and path pinning solutions that leverage non-proprietary, mature technologies over commodity Ethernet. Our evaluation follows a three-fold strategy, focusing on the key dimensions of performance, resiliency, and operational complexity. By applying this methodology to representative implementations, we highlight the trade-offs. While all techniques are resilient, path pinning-based solutions excel at performance but introduce greater complexity. Specifically, path pinning achieves up to 1.6× improvement over ECMP for RoCE test traffic and up to 2.5× for NCCL AllReduce. These results validate the promising performance benefits of path pinning and highlight the need to explore less complex implementations for broader adoption. Our methodology can be used to rigorously evaluate future implementations in support of AI network design.

Vela is a cloud-native system designed for LLM training workloads built using off-the-shelf hardware, Linux KVM-based virtualization, and a virtualized RDMA over Converged Ethernet (RoCE) network. Vela virtual machines (VMs) support peer-to-peer DMA between the GPUs and SRIOV-based network interface. In this paper, we share Vela's key architectural aspects with details from an NVIDIA A100 GPU-based deployment in one of the IBM Cloud data centers. Throughout the paper, we share insights and experiences from designing, building, and operating the system over a ~2.5 year timeframe to highlight the capabilities of readily available software and hardware technologies and the improvement opportunities for future AI systems, thereby making AI infrastructure more accessible to a broader community. As we evaluated the system for performance at ~1500 GPU scale, we achieved ~80% of the ideal throughput while training a 50 billion parameter decoder model using model parallelism, and ~70% per GPU FLOPS compared to a single VM with the High-Performance Linpack benchmark.

Modern Artificial Intelligence (AI) and HighPerformance Computing (HPC) workloads impose diverse demands on data center networks in terms of both performance and reliability. Applications such as inferencing are usually compute-bound, while distributed training and HPC workloads are network-bound, requiring high bandwidth and low, predictable latency. In terms of reliability, some workloads require seamless failure recovery, while others can tolerate failures through checkpointing or application/transport-layer reliability. In this paper, we address the challenge of designing a highperformance network for private data centers at scales of 1-2K Graphics Processing Units (GPUs) or other accelerators, supporting a range of workloads over Remote Direct Memory Access (RDMA), RDMA over Converged Ethernet (RoCE), and Transmission Control Protocol (TCP) with off-the-shelf hardware and software. We first present a network control plane design that accommodates flexible reliability and performance requirements, and we present our approach across key system components to efficiently utilize multiple equal-cost paths in a two-level leafspine topology. We then use simulations to guide our design choices, including a 1:4 speed ratio between server-facing and spine-facing ports, along with switch partitioning at the spine layer to mitigate the impact of flow collisions. Leveraging Ansible automation for efficient configuration management, we integrate these findings into a 12 -node cluster with 8 GPUs and 1.6 Tbps bandwidth per server. Using network micro-benchmarks that mimic the demands of intensive workloads, our results show that the network sustains near-line-rate throughput under traffic patterns where up to two-thirds of traffic traverses the spine, while also supporting a flexible reliability model.

Cloud networks support workloads with diverse characteristics. A key challenge facing cloud providers is how to meet the stringent performance and security needs of these diverse applications running over a shared infrastructure. To address this challenge, some providers build for peak capacity or even build dedicated clusters with specialized networks, which may be under-utilized at times. We propose a virtualization approach that customizes data center network resources to the needs of applications. Our approach is based on slicing data center network resources on-demand and customizing these slices to target workloads. Such slices can grow or shrink dynamically and programmatically based on workload demands. This elasticity provides a more efficient solution over building dedicated clusters. In our approach, a slice can be customized to a given set of workloads with similar security and performance requirements carved out of the underlying network. It leverages a software-defined underlay network controller and segment routing for fine-grained path control and service chaining. We have implemented a prototype of our fabric virtualization solution based on network slicing. In this paper, we first present the architecture of our prototype. Second, we present empirical results of slice provisioning times in networks of varying sizes and switch operating systems. Empirical results indicate that our prototype can support slice provisioning in the order of tens to hundreds of seconds and can meet the provisioning requirements of production networks.

We address energy-efficient placement of data and analytics components of composite analytics services on a wireless network to minimize execution-time energy consumption (computation and communication) subject to compute, storage and network resource constraints. We introduce an expressive analytics service hypergraph model for representing k-ary composability relationships (k ≥ 2) between various analytics and data components and leverage binary quadratic programming (BQP) to minimize the total energy consumption of a given placement of the analytics hypergraph nodes on the network subject to resource availability constraints. Then, after defining a potential energy functional Φ(·) to model the affinities of analytics components and network resources using analogs of attractive and repulsive forces in physics, we propose a decentralized Metropolis Monte Carlo (MMC) sampling method which seeks to minimize Φ by moving analytics and data on the network. Although Φ is non-convex, using a potential game formulation, we identify conditions under which the algorithm provably converges to a local minimum energy equilibrium placement configuration. Trace-based simulations of the placement of a deep-neural-network analytics service on a realistic wireless network show that for smaller problem instances our MMC algorithm yields placements with total energy within a small factor of BQP and more balanced workload distributions; for larger problems, it yields low-energy configurations while the BQP approach fails.

Hadoop has become the de facto standard for Big Data analytics, especially for workloads that use the MapReduce (M/R) framework. However, the lack of network awareness of the default MapReduce resource manager in Hadoop can cause unbalanced job scheduling, network bottleneck, and eventually increase the Hadoop run time if Hadoop nodes are clustered in several geographically distributed locations. In this paper, we present an application-aware network approach using software-defined networking (SDN) for distributed Hadoop clusters. We develop the SDN applications for this environment that consider network topology discovery, traffic monitoring, and flow rerouting in addition to loop avoidance mechanisms.