Large language models (LLMs) training or inference across multiple nodes introduces significant pressure on GPU memory and interconnect bandwidth. The Compute Express Link (CXL) shared memory pool offers a scalable solution by enabling memory sharing across nodes, reducing over-provisioning and improving resource utilization. We propose CXL-CCL, a collective communication library, leveraging the CXL shared memory pool to support cross-node GPU operations without relying on traditional RDMA-based networking. Our design addresses the challenges in synchronization, data interleaving, and communication parallelization faced by using the CXL shared memory pool for collective communications. Evaluating on multiple nodes with a TITAN-II CXL switch and six Micron CZ120 memory cards, we show that CXL-CCL achieves highly efficient collective operations across hosts, demonstrating CXL’s potential for scalable, memory-centric GPU communication. Our evaluation demonstrates that CXL-CCL achieves average performance improvements of 1.34 × for AllGather, 1.84 × for Broadcast, 1.94 × for Gather, and 1.07 × for Scatter, compared to the original RDMA-based implementation over 200 Gbps InfiniBand. In addition, an LLM training case study shows 1.11 × speedup compared with the InfiniBand while reducing interconnect hardware cost by 2.75 ×.

This paper presents TurboInfer, a system that enables power-efficient LLM inference on mobile devices. The core insight behind TurboInfer is that while LLMs are power-intensive due to their heavy computational demands, the model inference experiences unavoidable GPU stalls caused by tensor preparation for subsequent kernel executions at run-time. These GPU stalls arise from the unique host-controlled execution pipeline tailored to mobile phones and the significant DRAM access contention inherent to the shared memory architecture of mobile System-on-chips (SoCs). With LLM inference requiring hundreds to thousands of kernel executions, these short but frequent GPU stalls accumulate, accounting for over 74% of the token generation latency.

Virtual private clouds (VPCs) play a critical role in providing secure and isolated network environments for web services. However, with the growing number and size of VPCs, efficiently delivering control messages from the control plane to the data plane has become a major concern for cloud vendors. Existing end-to-end transmission solutions (e.g., RPC) will result in substantial overhead in the control plane, while message-oriented middleware-based solutions (e.g., message queue) will lead to high data plane overhead. To address this issue, we design Meteor, a high-performance control message delivery system for large-scale clouds. Specifically, Meteor combines an RPC path with a message queue (MQ) path and employs an auto dual-path switching mechanism to minimize the message delivery latency. Additionally, we propose a VPC-based message delivery and filtering scheme for the MQ path to reduce data plane overhead. We also design a delivery robustness guarantee mechanism to ensure the reachability and consistency of control messages. Meteor has been thoroughly tested with up to 100k container instances. Evaluation results show that Meteor decreases the message delivery latency by 48.8% and reduces the overhead by about 50% in real-world scenarios, compared with state-of-the-art solutions.

Building a realistic and scalable cloud emulator is a significant and unaddressed challenge in the cloud industry. The hyper-scale nature of modern cloud environments, encompassing millions of servers and network devices, combined with the complexity of multiple layers of network virtualization in both underlay and overlay networks, and the heterogeneity of devices from various vendors and models, makes this a complex task. This study develops a large-scale cloud emulator that supports the automatic and flexible configuration of both cloud underlay and overlay networks at scale. The underlay networks emulate various data center topologies and heterogeneous hardware, while the overlay networks emulate virtualized networks and associated private resources, such as networking, storage, services, and virtualization. In the end-to-end performance test, this proposed cloud emulator can emulate both emulated physical machines (EPMs) and emulated virtual machines (EVMs) in the underlay and overlay network, where 100,000 EVMs are emulated with a high emulation density of 10,000 EVMs per EPM in 20 mins. This study also showcases the emulation of a cloud data center in a 200-node Kubernetes cluster in AWS EC2, deploying 10,000 EPMs in a Clos network topology with 500,000 EVMs.

Driving southbound messages from a cloud control plane down to the distributed data plane on every compute node is one of the critical challenges in public clouds. Existing message delivery solutions solely based on remote procedure call (RPC) or message queue (MQ) tend to overlook strict resource constraints, e.g., network bandwidth and CPU capacity. This often results in extensive overhead in the control plane or message redundancy in the data plane, especially when a cloud receives highly concurrent user requests or experiences a rapid expansion. To this end, we design a dual-channel southbound message delivery framework, namely SMART, which combines an RPC channel with an MQ channel, to maximize the resource utilization in the cloud network. In the control plane, we implement a message parsing mechanism and propose a delivery channel selection algorithm based on the deep reinforcement learning (DRL) approach to support efficient dual-channel delivery under resource constraints. In the data plane, we design a message agent on each compute node to ensure the order preservation and state consistency of southbound messages. Both experimental and large-scale simulation results show that SMART demonstrates a reduction in control plane overhead by 64% compared to RPC and redundant messages by 45% compared to MQ, respectively.

Network administrators face the challenge of efficiently patching overwhelming volumes of vulnerabilities with limited time and resources. To address this issue, they must prioritize vulnerabilities based on the associated risk/severity measurements (i.e., CVSS). Existing solutions struggle to efficiently patch thousands of vulnerabilities on a network. This paper presents ILLATION, a proof-of-concept model that provides network-specific vulnerability risk prioritization to support efficient patching. ILLATION integrates AI techniques, such as neural networks and logical programming, to learn risk patterns from adversaries, vulnerability severity, and the network environment. It provides an integrated solution that learns and infers adversaries' motivation and ability in a network while also learning the constraints that restrict interactions between vulnerabilities and network elements. An evaluation of ILLATION against CVSS base and environmental metrics shows that it reflects changes in vulnerability scores and prioritization ranks as the same pattern as the CVSS model while identifying vulnerabilities with similar risk patterns to given adversaries better. On a simulated network with up to 10k vulnerable hosts and vulnerabilities, ILLATION can assess 1k vulnerabilities in about 4.5 minutes total, with an average running time of 0.19 seconds per vulnerability on a general-purpose computer.

The surging scale of distributed training (DT) incurs significant communication overhead in datacenters, while a promising solution is in-network aggregation (INA). It leverages programmable switches (e.g., Intel Tofino switches) for gradient aggregation to accelerate DT tasks. Due to switches’ limited on-chip memory size, existing solutions try to design the memory sharing mechanism for INA. This mechanism requires gradients to arrive at switches synchronously, while network dynamics make it common for the asynchronous arrival of gradients, resulting in existing solutions being inefficient (e.g., massive communication overhead). To address this issue, we propose GOAT, the first-of-its-kind work on gradient scheduling with collaborative in-network aggregation, so that switches can efficiently aggregate asynchronously arriving gradients. Specifically, GOAT first partitions the model into a set of sub-models, then decides which sub-model gradients each switch is responsible for aggregating exclusively and to which switch each worker should send its sub-model gradients. To this end, we design an efficient knapsack-based randomized rounding algorithm and formally analyze the approximation performance. We implement GOAT and evaluate its performance on a testbed consisting of 3 Intel Tofino switches and 9 servers. Experimental results show that GOAT can speed up the DT by$1.5 \times $compared to the state-of-the-art solutions.

Cloud infrastructure has gradually displayed a tendency of geographical distribution in order to provide anywhere, anytime connectivity to tenants all over the world. The tenant task placement in geo-distributed clouds comes with three critical and coupled factors:regional diversity in electricity prices,access delay for tenants, andtraffic demand among tasks. However, existing works disregard either the regional difference in electricity prices or the tenant requirements in geo-distributed clouds, resulting in increased operating costs or low user QoS. To bridge the gap, we design a cost optimization framework for tenant task placement in geo-distributed clouds, called TanGo. However, it is non-trivial to achieve an optimization framework while meeting all the tenant requirements. To this end, we first formulate the electricity cost minimization for task placement problem as a constrained mixed-integer non-linear programming problem. We then propose a near-optimal algorithm with a tight approximation ratio$(1-1/e)$using an effective submodular-based method. Results of in-depth simulations based on real-world datasets show the effectiveness of our algorithm as well as the overall 10%-30% reduction in electricity expenses compared to commonly-adopted alternatives.

Cloud infrastructure has gradually displayed a tendency of geographical distribution in order to provide anywhere, anytime connectivity to tenants all over the world. The tenant task placement in geo-distributed clouds comes with three critical and coupled factors: regional diversity in electricity prices, access delay for tenants, and traffic demand among tasks. However, existing works disregard either the regional difference in electricity prices or the tenant requirements in geo-distributed clouds, resulting in increased operating costs or low user QoS. To bridge the gap, we design a cost optimization framework for tenant task placement in geo-distributed clouds, called TanGo. However, it is non-trivial to achieve an optimization framework while meeting all the tenant requirements. To this end, we first formulate the electricity cost minimization for task placement problem as a constrained mixed-integer non-linear programming problem. We then propose a near-optimal algorithm with a tight approximation ratio (1 − 1/e) using an effective submodular-based method. Results of in-depth simulations based on real-world datasets show the effectiveness of our algorithm as well as the overall 10%-30% reduction in electricity expenses compared to commonly-adopted alternatives.

The surging scale of distributed training (DT) incurs significant communication overhead in datacenters, while a promising solution is in-network aggregation (INA). It leverages programmable switches (e.g., Intel Tofino switches) for gradient aggregation to accelerate the DT. Due to switches' limited on-chip memory size, existing solutions try to design the memory sharing mechanism for INA. This mechanism requires gradients to arrive at switches synchronously, while network dynamics make it common for the asynchronous arrival of gradients, resulting in existing solutions being inefficient (e.g., massive communication overhead). To address this issue, we propose GOAT, the first-of-its-kind work on gradient scheduling with collaborative in-network aggregation, so that switches can efficiently aggregate asynchronously arriving gradients. Specifically, GOAT first partitions the model into a set of sub-models, then decides which sub-model gradients each switch is responsible for aggregating exclusively and to which switch each worker should send its sub-model gradients. To this end, we design an efficient knapsack-based randomized rounding algorithm and formally analyze the approximation performance. We implement GOAT and evaluate its performance on a testbed consisting of 3 Intel Tofino switches and 9 servers. Experimental results show that GOAT can speed up the DT by 1.5× compared to the state-of-the-art solutions.

With the broad deployment of distributed applications on clouds, east-west traffic is now dominating the majority of cloud networks. The existing communication solutions are tightly coupled with either the control plane (e.g., preprogrammed model) or the location of compute nodes (e.g., conventional gateway model). As a result, it is difficult to flexibly respond to the rapidly expanding networks and frequent abnormal events (e.g., burst traffic and device failures). Accordingly, they may not provide high-performance east-west forwarding while ensuring scalability and robustness. To address this issue, we design Zeta, a scalable and robust east-west forwarding framework with gateway clusters for hyperscale clouds. Zeta abstracts the traffic forwarding capability as a Gateway Cluster Layer, decoupled from the logic of control plane and the location of compute nodes. Specifically, Zeta adopts gateway clusters to support large-scale networks and cope with burst traffic. Moreover, a transparent Multi IPs Migration is proposed for fast recovery from unpredictable failures. We implement Zeta based on eXpress Data Path (XDP) and evaluate its scalability and robustness through comprehensive experiments with up to 100k container instances. Our evaluation shows that Zeta reduces the 99% RTT by$5.1 {\times }$in burst video traffic, and reduces the gateway pure recovery delay by$10.8 {\times }$compared with the state-of-the-art solutions.

Southbound message delivery from the control plane to the data plane is one of the essential issues in multi-tenant clouds. A natural method of southbound message delivery is that the control plane directly communicates with compute nodes in the data plane. However, due to the large number of compute nodes, this method may result in massive control overhead. The Message Queue (MQ) model can solve this challenge by aggregating and distributing messages to queues. Existing MQ-based solutions often perform message aggregation based on the physical network topology, which do not align with the fundamental requirements of southbound message delivery, leading to high message redundancy on compute nodes. To address this issue, we design and implement VITA, the first-of-its-kind work on virtual network topology-aware southbound message delivery. However, it is intractable to optimally deliver southbound messages according to the virtual attributes of messages. Thus, we design two algorithms, submodular-based approximation algorithm and simulated annealing-based algorithm, to solve different scenarios of the problem. Both experiment and simulation results show that VITA can reduce the total traffic amount of redundant messages by 45%-75% and reduce the control overhead by 33%-80% compared with state-of-the-art solutions.

Traditional kernel network processing suffers from high delay and overhead, which has become the bottleneck of high-speed networks. A natural method to accelerate packet processing is to bypass the kernel network stack and process packets in user space directly, $e.g$., DPDK. However, due to many network functions are implemented in the kernel network stack, bypassing the stack means that we need to redesign the required functions elsewhere, leading to poor compatibility. One promising technology to address this problem is called eXpress Data Path (XDP), which can support high-performance packet processing while preserving the kernel stack. However, existing solutions mainly run XDP in software mode, resulting in relatively poor packet processing performance. Fortunately, with the development of programmable hardware, running XDP in hardware mode is a more promising approach. Thus, in this paper, we design and implement OXDP, the first-of-its-kind work on accelerating packet processing by offloading XDP to SmartNICs. Since today’s SmartNICs are still subject to some limitations regarding the rigid runtime environment, it is nontrivial to offload XDP to SmartNICs. To address this issue, OXDP performs best-effort offloading based on the primitive packet operations, thus maximizing the use of SmartNIC’s resources. Specifically, OXDP splits the forwarding function into two parts, one part offloading on SmartNIC with hardware XDP and the other part deploying on host. We evaluate the efficiency of OXDP with comprehensive experiments. Evaluation results show that the forwarding rate of OXDP can reach 18.7 Mpps, which improves $30 \times$ compared with the single-core performance of software XDP.

In a hyper-scale cloud data center, a large number of control messages are distributed to hundreds of thousands of compute nodes from a logically centralized control plane. The delivery of these control messages, a.k.a. southbound messages, is critical to cloud infrastructure management, as it greatly affects customer experience. Existing works mainly deal with southbound message delivery by two methods: point-to-point transmission and Message Queue (MQ)-based solutions. With the point-to-point transmission method, each message is sent from the control plane to compute nodes directly, which may cause high control complexity and overhead in hyper-scale clouds. The MQ-based method can address the challenge of high complexity through message aggregation and subscribe/publish model. However, it usually brings in redundant messages, and further causes extra load on compute nodes. To solve the problem above, we design SNIP, which exploits the ability of programmable switches to perform in-network message pruning and to reduce message redundancy. Specffically, forwarding and processing information computed by the control plane is attached to the package header of every control message. Redundant messages can be identified and processed by programmable switches. In addition, we propose a rounding-based algorithm to prune messages with minimal redundancy. The simulation results show that SNIP can reduce the control overhead by 80%-85% and the total traffic of redundant messages by 35% compared with existing solutions.

Southbound message delivery from the control plane to the data plane is one of the essential issues in multi-tenant clouds. A natural method of southbound message delivery is that the control plane directly communicates with compute nodes in the data plane. However, due to the large number of compute nodes, this method may result in massive control overhead. The Message Queue (MQ) model can solve this challenge by aggregating and distributing messages to queues. Existing MQ-based solutions often perform message aggregation based on the physical network topology, which do not align with the fundamental requirements of southbound message delivery, leading to high message redundancy on compute nodes. To address this issue, we design and implement VITA, the first-of-its-kind work on virtual network topology-aware southbound message delivery. However, it is intractable to optimally deliver southbound messages according to the virtual attributes of messages. Thus, we design two algorithms, submodular-based approximation algorithm and simulated annealing-based algorithm, to solve different scenarios of the problem. Both experiment and simulation results show that VITA can reduce the total traffic amount of redundant messages by 45%-75% and reduce the control overhead by 33%-80% compared with state-of-the-art solutions.

Wireless energy transfer (WET) is a new technology that can be used to charge the batteries of sensor nodes without wires. Although wireless, WET does require a charging station to be brought to within reasonable range of a sensor node so that a good energy transfer efficiency can be achieved. On the other hand, it has been well recognized that data collection with a mobile base station has significant advantages over a static one. Given that a mobile platform is required for WET, a natural approach is to employ the same mobile platform to carry the base station for data collection. In this paper, we study the interesting problem of co-locating a wireless charger (for WET) and a mobile base station on the same mobile platform-the wireless charging vehicle (WCV). The WCV travels along a pre-planned path inside the sensor network. Our goal is to minimize energy consumption of the entire system while ensuring that 1) each sensor node is charged in time so that it will never run out of energy, and 2) all data collected from the sensor nodes are relayed to the mobile base station. We develop a mathematical model for this problem (OPT-t), which is time-dependent. Instead of solving OPT-t directly, we show that it is sufficient to study a special subproblem (OPT-s) which only involves space-dependent variables. Subsequently, we develop a provably near-optimal solution to OPT-s. Our results offer a solution on how to use a single mobile platform to address both WET and data collection in sensor networks.

Wireless energy transfer based on magnetic resonant coupling is a promising technology to replenish energy to a wireless sensor network (WSN). However, charging sensor nodes one at a time poses a serious scalability problem. Recent advances in magnetic resonant coupling show that multiple nodes can be charged at the same time. In this paper, we exploit this multi-node wireless energy transfer technology and investigate whether it is a scalable technology to address energy issues in a WSN. We consider a wireless charging vehicle (WCV) periodically traveling inside a WSN and charging sensor nodes wirelessly. Based on charging range of the WCV, we propose a cellular structure that partitions the two-dimensional plane into adjacent hexagonal cells. We pursue a formal optimization framework by jointly optimizing traveling path, flow routing, and charging time. By employing discretization and a novel Reformulation-Linearization Technique (RLT), we develop a provably near-optimal solution for any desired level of accuracy. Through numerical results, we demonstrate that our solution can indeed address the charging scalability problem in a WSN.

Wireless energy transfer is a promising technology to fundamentally address energy and lifetime problems in a wireless sensor network (WSN). On the other hand, it has been well recognized that a mobile base station has significant advantages over a static one. In this paper, we study the interesting problem of co-locating the mobile base station on the wireless charging vehicle (WCV). The goal is to minimize energy consumption of the entire system while ensuring none of the sensor nodes runs out of energy. We develop a mathematical model for this complex problem. Instead of studying the general problem formulation (OPT-t), which is time-dependent, we show that it is sufficient to study a special subproblem (OPT-s) which only involves space-dependent variables. Subsequently, we develop a provably near-optimal solution to OPT-s. The novelty of this research mainly resides in the development of several solution techniques to tackle a complex problem that is seemingly intractable at first glance. In addition to addressing a challenging and interesting problem in a WSN, we expect the techniques developed in this research can be applied to address other related networking problems involving time-dependent movement, flow routing, and energy consumption.

Wireless power transfer is a promising technology to fundamentally address energy problems in a wireless sensor network. To make such a technology work effectively, a vehicle is needed to carry a charger to travel inside the network. On the other hand, it has been well recognized that a mobile base station offers significant advantages over a fixed one. In this paper, we investigate an interesting problem of co-locating the mobile base station on the wireless charging vehicle. We study an optimization problem that jointly optimizes traveling path, stopping points, charging schedule, and flow routing. Our study is carried out in two steps. First, we study an idealized problem that assumes zero traveling time, and develop a provably near-optimal solution to this idealized problem. In the second step, we show how to develop a practical solution with non-zero traveling time and quantify the performance gap between this solution and the unknown optimal solution to the original problem.

Wireless energy transfer based on magnetic resonant coupling is a promising technology to replenish energy to sensor nodes in a wireless sensor network (WSN). However, charging sensor node one at a time poses a serious scalability problem. Recent advances in magnetic resonant coupling shows that multiple nodes can be charged at the same time. In this paper, we exploit this multi-node wireless energy transfer technology to address energy issue in a WSN. We consider a wireless charging vehicle (WCV) periodically traveling inside a WSN and charging sensor nodes wirelessly. We propose a cellular structure that partitions the two-dimensional plane into adjacent hexagonal cells. The WCV visits these cells and charge sensor nodes from the center of a cell. We pursue a formal optimization framework by jointly optimizing traveling path, flow routing and charging time. By employing discretization and a novel Reformulation-Linearization Technique (RLT), we develop a provably near-optimal solution for any desired level of accuracy.

Wireless sensor networks are constrained by limited battery energy. Thus, finite network lifetime is widely regarded as a fundamental performance bottleneck. Recent breakthrough in the area of wireless power transfer offers the potential of removing this performance bottleneck, i.e., allowing a sensor network to remain operational forever. In this paper, we investigate the operation of a sensor network under this new enabling energy transfer technology. We consider the scenario of a mobile charging vehicle periodically traveling inside the sensor network and charging each sensor node's battery wirelessly. We introduce the concept of renewable energy cycle and offer both necessary and sufficient conditions. We study an optimization problem, with the objective of maximizing the ratio of the wireless charging vehicle (WCV)'s vacation time over the cycle time. For this problem, we prove that the optimal traveling path for the WCV is the shortest Hamiltonian cycle and provide a number of important properties. Subsequently, we develop a near-optimal solution by a piecewise linear approximation technique and prove its performance guarantee.

Traditional wireless sensor networks are constrained by limited battery energy. Thus, finite network lifetime is widely regarded as a fundamental performance bottleneck. Recent breakthrough in the area of wireless energy transfer offers the potential of removing such performance bottleneck, i.e., allowing a sensor network remain operational forever. In this paper, we investigate the operation of a sensor network under this new enabling energy transfer technology. We consider the scenario of a mobile charging vehicle periodically traveling inside the sensor network and charging each sensor node's battery wirelessly. We introduce the concept of renewable energy cycle and offer both necessary and sufficient conditions. We study an optimization problem, with the objective of maximizing the ratio of the wireless charging vehicle (WCV)'s vacation time over the cycle time. For this problem, we prove that the optimal traveling path for the WCV is the shortest Hamiltonian cycle and provide a number of important properties. Subsequently, we develop a near-optimal solution and prove its performance guarantee.