The rapid explosion of Artificial Intelligence (AI) workloads utilizing a growing number of accelerators has placed unprecedented demand on the network. These workloads typically leverage Remote Direct Memory Access (RDMA) and require a high-performance network fabric. While many purpose-built cloud networking solutions can provide high performance, efficiently utilizing these costly infrastructures require a fabric that is multi-tenant for ease of consumption. Furthermore, the fabric must be resilient to faults and for operational manageability. Resilient cloud networks typically employ mature Ethernet segmentation techniques over Clos topologies with Equal Cost Multi Path (ECMP) routing. ECMP hashes flows to paths, which in case of collisions can significantly degrade performance for large RDMA over Converged Ethernet (RoCE) flows. To mitigate ECMP penalties, we evaluate routing strategies with varying levels of operational complexity. We explore load balancing and path pinning solutions that leverage non-proprietary, mature technologies over commodity Ethernet. Our evaluation follows a three-fold strategy, focusing on the key dimensions of performance, resiliency, and operational complexity. By applying this methodology to representative implementations, we highlight the trade-offs. While all techniques are resilient, path pinning-based solutions excel at performance but introduce greater complexity. Specifically, path pinning achieves up to 1.6× improvement over ECMP for RoCE test traffic and up to 2.5× for NCCL AllReduce. These results validate the promising performance benefits of path pinning and highlight the need to explore less complex implementations for broader adoption. Our methodology can be used to rigorously evaluate future implementations in support of AI network design.

Vela is a cloud-native system designed for LLM training workloads built using off-the-shelf hardware, Linux KVM-based virtualization, and a virtualized RDMA over Converged Ethernet (RoCE) network. Vela virtual machines (VMs) support peer-to-peer DMA between the GPUs and SRIOV-based network interface. In this paper, we share Vela's key architectural aspects with details from an NVIDIA A100 GPU-based deployment in one of the IBM Cloud data centers. Throughout the paper, we share insights and experiences from designing, building, and operating the system over a ~2.5 year timeframe to highlight the capabilities of readily available software and hardware technologies and the improvement opportunities for future AI systems, thereby making AI infrastructure more accessible to a broader community. As we evaluated the system for performance at ~1500 GPU scale, we achieved ~80% of the ideal throughput while training a 50 billion parameter decoder model using model parallelism, and ~70% per GPU FLOPS compared to a single VM with the High-Performance Linpack benchmark.

Modern Artificial Intelligence (AI) and HighPerformance Computing (HPC) workloads impose diverse demands on data center networks in terms of both performance and reliability. Applications such as inferencing are usually compute-bound, while distributed training and HPC workloads are network-bound, requiring high bandwidth and low, predictable latency. In terms of reliability, some workloads require seamless failure recovery, while others can tolerate failures through checkpointing or application/transport-layer reliability. In this paper, we address the challenge of designing a highperformance network for private data centers at scales of 1-2K Graphics Processing Units (GPUs) or other accelerators, supporting a range of workloads over Remote Direct Memory Access (RDMA), RDMA over Converged Ethernet (RoCE), and Transmission Control Protocol (TCP) with off-the-shelf hardware and software. We first present a network control plane design that accommodates flexible reliability and performance requirements, and we present our approach across key system components to efficiently utilize multiple equal-cost paths in a two-level leafspine topology. We then use simulations to guide our design choices, including a 1:4 speed ratio between server-facing and spine-facing ports, along with switch partitioning at the spine layer to mitigate the impact of flow collisions. Leveraging Ansible automation for efficient configuration management, we integrate these findings into a 12 -node cluster with 8 GPUs and 1.6 Tbps bandwidth per server. Using network micro-benchmarks that mimic the demands of intensive workloads, our results show that the network sustains near-line-rate throughput under traffic patterns where up to two-thirds of traffic traverses the spine, while also supporting a flexible reliability model.

Cloud networks support workloads with diverse characteristics. A key challenge facing cloud providers is how to meet the stringent performance and security needs of these diverse applications running over a shared infrastructure. To address this challenge, some providers build for peak capacity or even build dedicated clusters with specialized networks, which may be under-utilized at times. We propose a virtualization approach that customizes data center network resources to the needs of applications. Our approach is based on slicing data center network resources on-demand and customizing these slices to target workloads. Such slices can grow or shrink dynamically and programmatically based on workload demands. This elasticity provides a more efficient solution over building dedicated clusters. In our approach, a slice can be customized to a given set of workloads with similar security and performance requirements carved out of the underlying network. It leverages a software-defined underlay network controller and segment routing for fine-grained path control and service chaining. We have implemented a prototype of our fabric virtualization solution based on network slicing. In this paper, we first present the architecture of our prototype. Second, we present empirical results of slice provisioning times in networks of varying sizes and switch operating systems. Empirical results indicate that our prototype can support slice provisioning in the order of tens to hundreds of seconds and can meet the provisioning requirements of production networks.

Performance and scalability of cloud applications are heavily dependent on those same characteristics in the underlying network. Several factors complicate cloud network evaluation. Most notable are 1) large scale, 2) a practically infinite set of application workloads, 3) a large set of potential metrics, 4) a large variety of available configurations within and across clouds and 5) a lack of visibility into the underlying network. There is a tremendous need for a cloud network evaluation methodology that overcomes these challenges, that is practical to apply and provides predictive results. We describe our work to develop such a methodology for cloud network evaluation and a tool (Euclid) that automates its application. Euclid leverages a set of widely used traffic generators to evaluate throughput and latency across a cloud network between endpoint pairs. Evaluation of these micro benchmarks between and among different sets of endpoints is coordinated to reflect several well-known application traffic patterns. Large sets of results are distilled into a tractable set of metrics and associated graphs that facilitate consumption of the results by cloud operators and users. We report results from applying Euclid to seventeen cloud configurations that span three public and one private cloud. The results demonstrate the practicality of our approach and highlight some interesting differences across clouds.

Growth in cloud computing motivates cloud networks that provide excellent performance and scalability. Improvements rely on the ability to measure these characteristics. Measurement is complicated by a combinatorial explosion of implementations, configurations, metrics, workloads and scenarios. We present tools and a framework that facilitate performance and scalability evaluation. We present the results of applying the framework to six OpenStack network implementations/configurations. The results validate the framework's ability to highlight the performance and scalability impact of changes to the underlying cloud implementation and configuration. Our experience also yields important lessons regarding cloud network evaluation.

In this work, we study the problem of efficient resource selection in real-time distributed systems. Specifically, we study how a client in the presence of measurement uncertainty can rapidly select a resource (target) that exhibits good network connectivity. The problem has many applications, including peer selection in P2P systems, server/gateway selection in IP telephony, and overlay routing. A probing-based search strategy is complicated by the potentially large number of targets and the inherent uncertainty in measurement results due to, for instance, small measurement sample sizes and other forms of "noise." We parameterize the problem based on the number of available targets, the number of "good" targets, and various network characteristics. We introduce a framework called spatial slow start (SSS) for searching the target space. SSS can be considered as a parameterized tree search with an interesting twist: apart from the target search space, the rate of probing per target is also manipulated at each step. A detailed simulation study provides insight on what SSS choices are significant under different network conditions. The simulation study also establishes that previously proposed techniques, namely, a random selection (RAND) algorithm that does not do any probing, and a "quick probe and select the best" (QPS) technique work well, but only in some situations. Based on these insights, we present an adaptive SSS algorithm that not only guides the search process according to its measurements of various system parameters, but also changes its search parameters in response to the quality of the measurements themselves. This new adaptive SSS algorithm performs as well as RAND and QPS in situations where RAND and QPS are efficient, and outperforms them in other cases.

Network topology information has many uses for networked applications including the design of reliability and performance enhancing schemes. This information is often acquired by network discovery techniques that rely on ICMP and/or SNMP support from the infrastructure. Unfortunately, network infrastructure may support such services at varying levels. In cases where such support is limited, techniques based on strictly end-to-end measurements have been proposed in the literature to infer a logical topology. In this paper, we propose an end-to-end measurement technique that is complimentary to the earlier techniques and relies on multiple-source multiple-destination probing to collect network wide measurements faster. Our technique relies on the interaction between streams of packets at various nodes in the network. Specifically, we aim to measure the reaction of end-to-end probes to a short burst of concentrated traffic (signal) injected to the network. The key assumption is that probes traversing paths that physically intersect with the path of the bursty traffic (signal path) will exhibit extra jitter. The occurrence of extra jitter in the probe packet inter-arrival times matching the signal bursts indicates that the paths of the probe and signal intersect. We conducted an empirical study on an actual production network to evaluate the ability of the technique to detect end-to-end path intersections. Our empirical results indicate that in the network we considered the technique successfully detected path intersections.

Application-layer path switching systems ensure the performance and reliability of real-time applications (such as IP Telephony) by exploiting inherent redundancies in the underlying IP network. These systems monitor quality throughout an IP Telephony call, rapidly detect performance problems and, if necessary, re-direct calls around problems and congestion in the network. Under normal network load conditions, these systems can easily find a feasible alternate path and restore call quality within sub-second timeframes. However, at high network load, finding alternate paths is challenging due to inherent inaccuracies in the techniques that attempt to measure an end-to-end path’s ability to accommodate additional calls. These inaccuracies may adversely impact established calls on the path and some of the calls may even oscillate back and forth between various alternate paths. In this paper, we propose some novel techniques that address these problems and ensure system stability and performance at high network load. We introduce the concept of Spatial Slow Start, which allows rapid selection of a feasible path from a large possible set. In addition, we help protect a path’s established calls from other switching calls by using intelligent back-off and probationary admission control techniques. We conducted extensive simulation studies to evaluate the performance of the proposed techniques. The results indicate that our techniques significantly reduce the number of oscillations and the amount of time that calls experience poor quality.

An important consideration when deploying realtime applications such as IP Telephony is how to ensure that the network delivers the stringent QoS requirements of these applications. Prior to deployment, assessing whether an enterprise network can handle IP Telephony ensures a successful deployment. A network assessment approach recently introduced in the ExpertNetTMtool suite involves distributing small measurement devices in the network. These devices inject synthetic RTP traffic and collect end-to-end measurements that represent end-to-end call quality. In this approach, the placement of measurement devices impacts the network coverage and accuracy of the analysis. In this paper, we address the problem of how to place such measurement devices in a network to ensure a comprehensive analysis. Specifically, we focus on two issues: first where to place network measurement devices when the network topology is known and second how to estimate the number of such devices in the absence of network topology information. In this paper, we address the first problem by providing two heuristic algorithms for placing measurement devices in a given network. We evaluate these heuristics using synthetic networks. We address the second problem by proposing a technique to estimate the number of measurement devices using the size of a network. To evaluate the performance of our technique, we conducted a study on actual enterprise network topologies. Using our technique, we predicted the number of endpoints based on network size and compared these figures to the number of endpoints placed by the proposed algorithms using the network topology. Our results indicate that the predictions are comparable to the number of endpoints placed using the actual enterprise network topology.

Application-layer solutions to improve the reliability of realtime applications such as IP telephony exploit the inherent redundancy in networks. Earlier work has shown that the ISP networks provide sufficient redundancy that can be used by such overlay solutions. Enterprise networks, on the other hand, are designed differently from ISP networks, and implementing an overlay on enterprise networks introduces additional constraints. In this paper, we explore the level of redundancy in enterprise networks that would justify an application-layer reliability solution, while taking into account the unique characteristics of enterprise deployments. We report the findings of an empirical study we conducted on nine real enterprise networks of varying sizes. Specifically, we quantify the amount of redundancy in enterprise networks and the protection level that can be achieved by single-hop versus multi-hop overlays. Our results indicate that, among the networks we considered, on average 30% and up to 86% of the nodes on end-to-end paths can be protected. Furthermore, in these networks going from a single-hop overlay to a multiple-hop overlay provided only marginal benefits.

Maintaining good quality of service for real-time applications like IP Telephony requires quick detection and reaction to network impairments. In this paper, we propose and study novel measurement techniques in ORBIT, which is a simple, easily deployable architecture that uses single-hop overlays implemented with intelligent endpoints and independent relays. The measurement techniques provide rapid detection and recovery of IP Telephony during periods of network trouble. We study our techniques via detailed simulations of several multi-site enterprise topologies of varying sizes and three typical fault models. We show that our proposed techniques can detect network impairments rapidly and rescue IP Telephony calls in sub-second intervals. We observed that all impacted calls were rescued with only a few relays in the network and the run-time overhead was low. Furthermore, the relay sites needed to be provisioned with minimal additional bandwidth to support the redirected calls.

Multimedia applications such as IP telephony are among the applications that demand strict quality of service (QoS) guarantees from the underlying data network. At the predeployment stage it is critical to assess whether the network can handle the QoS requirements of IP telephony and fix problems that may prevent a successful deployment. In this paper we describe a technique for efficiently assessing network readiness for IP telephony. Our technique relies on understanding link level QoS behavior in a network from an IP telephony perspective. We use network topology and end-to-end measurements collected from the network in locating the sources of performance problems that may prevent a successful IP telephony deployment. We present an empirical study conducted on a real network spanning three geographically separated sites of an enterprise network. The empirical results indicate that our approach efficiently and accurately pinpoints links in the network incurring the most significant delay.

Networked multimedia applications require stringent quality of service (QoS) guarantees from the underlying data network. Many techniques have been proposed at the network layer to deliver acceptable QoS for such applications. In this paper, we explore providing network monitoring support in the application layer for QoS policies. We study the feasibility of a scalable network monitoring service that real-time applications may access to gather network status information in order to adapt to changing network conditions and take actions such as making application level routing decisions. We propose a scalable, low probing overhead monitoring service based on monitoring utilization on network devices traversed by multimedia traffic. Our motivation is to take advantage of often intersecting end-to-end paths traversed by multimedia traffic in an enterprise setting. We empirically assess the QoS improvements resulting from a service that monitors network utilization for an application involving dynamic server selection on a real production network. Specifically, we compare the performance of various dynamic server selection strategies for multimedia streaming including random selection, round-trip time based selection, proximity based selection, and utilization based selection. Our results indicate that compared to random selection, server selection strategies that rely on our network utilization based monitoring scheme result in significantly lower end-to-end delay and packet loss in streaming sessions. Compared to round-trip time based selection, our scheme results in comparable end-to-end delay and loss figures while incurring significantly lower probing cost

Networked multimedia applications require stringent real-time QoS guarantees. Successful deployment of such applications closely depends on the performance of the underlying data network. T he characteristics and the QoS requirements of these applications are different from traditional data applications. Hence, prior to deployment it is necessary to evaluate a network from a multimedia perspective. In this paper, we focus on IP Telephony and describe a framework for providing tools for IP Telephony readiness evaluation. This framework can be easily generalized to other multimedia applications. Our approach injects synthesized voice traffic and measures perceived end-to-end quality. We present a novel idea of relating voice quality metrics to the performance of data network devices. Following the proposed framework, we developed a prototype tool to evaluate a network and to identify problems, if any, prior to IP Telephony deployment. Our tool automatically discovers the topology of a given network, and collects and integrates network device performance and voice quality metrics. We describe the architecture of our tool and provide sample outputs from a network consisting of 129 devices.

P. Godefroid (1997) developed a state-space exploration technique that does not store states in memory. This stateless search technique is effective for testing concurrent programs. It performs deterministic executions of a concurrent program by automatically controlling the execution of synchronization operations. This technique also uses partial order reduction methods to reduce the chance of executing two or more different totally-ordered paths with the same partial order. In this paper, we propose extensions to Godefroid's technique and compare it with other stateless search techniques for testing concurrent programs. Our empirical results indicate that a combination of stateless searching and a simple cycle prediction method is very cost-effective for detecting deadlocks and violations of assertions.

Simultaneous reachability analysis (SRA) is a recently proposed technique to alleviate the state space explosion problem in reachability analysis of concurrent systems. Its goal is to reduce the number of generated states while guaranteeing the detection of certain types of faults in the system such as deadlock and unexecutable transitions. The main idea of SRA is to allow a global transition in a reachability graph to contain a set of local transitions (i.e. transitions of individual processes) such that the state reached by the global transition is independent of the execution order of the associated local transitions. In this paper, we show how to apply the SRA approach to systems modeled as extended finite state machines (EFSM) with multiple ports. Empirical results from applying our SRA algorithm to the dining philosophers problem indicate that our algorithm reduces the number of generated states and the computation time by about 90%.