Modern NICs DMA packets directly to the LLC using technologies like DDIO, reducing access latencies for networking applications. Prior work has observed several performance issues with DDIO when the working set of packet buffers does not fit into LLC. For example, the leaky DMA problem arises when incoming packets evict older packets that have not yet been processed by the application from LLC, causing them to be fetched again from main memory. While using a smaller pool of packet buffers that fits in cache is an obvious solution, this may result in the NIC running out of buffers to DMA packets into when a burst of packets arrives. This paper proposes Toasty, a system that mitigates this tradeoff between high throughput and resilience to packet loss that arises when sizing the network packet buffer pool. While prior work has proposed hardware-based solutions to this problem, Toasty is a software-only solution that can be deployed on commodity NIC hardware. Toasty manages the packet buffer pool as a LIFO stack instead of a FIFO queue, and adapts the number of buffers populated into the NIC hardware RX ring based on incoming packet load and application processing rate. Together, these changes enable Toasty to recirculate a small working set of cache-warm buffers in steady state, while falling back to a larger pool of buffers during traffic bursts. We implement Toasty over the AF_XDP kernel bypass framework, and our evaluation shows that Toasty improves network throughput for a variety of network functions by up to 78% over the default buffer pool implementation of AF_XDP. We also show that Toasty matches the performance of a small buffer pool that fits in cache, while being more resilient to traffic bursts.

Recent advances in the Linux kernel, such as eXpress Data Path (XDP) and AF_XDP sockets, enable high-speed packet processing for software NFs while preserving access to kernel features. However, the default AF_XDP implementation in the Linux kernel does not permit easy and performant NF chaining, e.g., zero-copy transfer of packets across NFs co-located on the same host. While prior work has proposed solutions for optimized NF chaining in the context of kernel bypass frameworks like DPDK that operate entirely in userspace, such solutions do not extend easily to AF_XDP, because the AF_XDP datapath is fragmented across the kernel driver and userspace. This paper introduces FLASH, a low-overhead inkernel chaining mechanism for AF_XDP sockets. FLASH enables zero-copy packet transfers for FLASH-native NFs, and single-copy packet transfers for legacy AF_XDP NFs. Further, via integration with K8s, FLASH supports the deployment of unprivileged containerized NFs on cloud platforms. Our work contributes several novel modifications to the AF_XDP datapath in the kernel to implement optimized NF chaining, and provides userspace libraries/APIs to easily build NFs that leverage FLASH. Our evaluations show that FLASH matches the performance of userspace DPDK-based NF chaining frameworks, while outperforming the best available AF_XDP-based alternatives by up to 2.5× in throughput, and achieving the lowest latency among all NF chaining frameworks.

Text-based specifications are the de-facto standard for specifying complex multi-tier systems. For example, 3GPP specifications define various interfaces, messages, and message processing at the multiple inter-connected nodes of a 5G system. These standards documents tend to be verbose, and may be ambiguous or inconsistent in places, increasing programmer effort to implement them in a general purpose language. This paper presents Pyramis, a Domain Specific Language (DSL) with suitable high-level abstractions for specifying the interfaces, messages, and processing in a multi-tier system. Pyramis allows programmers to specify multi-tier systems in a concise and precise manner, and enables easy development of software based on the specifications. We also develop a translator with Pyramis that automatically generates optimized, multi-threaded C++ code for the various components of the multi-tier system from the specification, and also generates eBPF-based measurement code for computing various performance metrics. We use Pyramis to build several components in the 5G mobile packet core. We show that the specifications written in Pyramis are 2–3 × smaller than the actual reference implementation, while the auto-generated C++ code performs on par with a hand-optimized implementation. We believe that Pyramis can eventually replace verbose text specifications like the 3GPP standards documents in telecom systems.

Efforts to improve multicore scalability of network functions (NFs) have traditionally focused on making network stacks scalable via partitioning TCP/IP data structures into per-core slices and ensuring flow-to-core affinity, leading to elimination of locking in the network stack while processing an incoming packet. But the above techniques fail to eliminate locking in NFs which store state at the granularity of an application-layer key that does not map to a TCP/IP flow, e.g., NFs in the 5G packet core that store state at the granularity of a mobile subscriber/user, where requests from a user could arrive over multiple flows, or requests from multiple users can arrive on a single flow. Prior work does not allow steering all traffic of a particular user to the same core for such NFs. This paper presents AppSteer, a framework that enables application-aware steering of incoming requests to cores for NFs running on the Linux kernel, in order to localize the requests of a given application-layer entity (e.g., mobile user) to a single core. NFs running over AppSteer can then partition their state into per-core slices and access it in a lockfree manner, leading to better multicore scalability. We evaluate AppSteer by building lockfree versions of production-grade 5G core NFs running on top of AppSteer and show that they have 15–18% higher throughput at 16 cores when compared to their locking-based counterparts.

This paper presents DDIOSim, a cycle-accurate microarchitecture simulator that simulates Data Direct I/O-based network packet processing. Our open-source simulator consists of a front-end trace generator that generates traces of CPU instructions, memory accesses, and network I/O events across multiple networking applications, and a cycle-accurate backend simulator that processes these traces by simulating DDIO operations along with the entire CPU and cache/memory hierarchy. Our simulator can be used to explore various DDIO design and configuration options, and its interactions with other microarchitecture optimizations like cache hierarchy, hardware prefetchers, and DRAM schedulers.

Emerging 5G applications require a dataplane that has a high forwarding throughput and low processing latency, in addition to low cost and power consumption. To meet these requirements, the state-of-the-art 5G User Plane Functions (UPFs) are built over high performance packet I/O mechanisms like the Data Plane Development Kit (DPDK), and further offload some functionality to programmable dataplane hardware. In this paper, we design and implement several standards-compliant UPF prototypes, beginning with a software-only DPDK-based UPF, progressing to designs which offload different functions to programmable hardware. We evaluate and compare the performance of these designs, to highlight the costs and benefits of these offloads. Our results show that offload techniques employed in prior work help improve performance in certain scenarios, but also have their limitations. Overcoming these limitations and fully realizing the power of programmable hardware requires offloading more complex functionality than is done today. Our work presents a preliminary implementation towards a comprehensive programmable dataplane-accelerated 5G UPF.

Several novel userspace network stacks have been proposed in recent research to overcome the limitations of the Linux network stack in providing high-performance I/O for Virtual Network Functions (VNFs). In this paper, we evaluate the performance of several state-of-the-art network stacks in the context of the VNFs of the 5G mobile packet core. The VNFs in the 5G core are several times more compute-intensive than the VNFs used to benchmark network stacks in prior work, given the need to perform user authentication and other such cryptographic operations. Our evaluation shows that while modern stacks outperform the Linux kernel stack over I/O intensive VNFs (as observed in prior work), the performance gap is not as wide in the case of CPU-intensive VNFs of the 5G core. We also find that the packet core VNFs can obtain up to 67% higher performance if the network stack could partition traffic to CPU cores at the granularity at which VNFs maintain state (mobile subscriber in this case), enabling a lockfree architecture within the VNF. The insights from our work can help us design a network stack that is better suited for compute-intensive VNFs such as those in the 5G core.

Network Function Virtualization (NFV) aims to reduce costs and increase flexibility of networks by moving functionality traditionally implemented in custom hardware into software packet processing applications, or virtual network functions (VNFs), running on commodity servers in a cloud. This paper describes the design and implementation of libVNF, a library to build high performance, horizontally scalable VNFs. Unlike existing frameworks for VNF development, our library (i) can be used for the development of L2/L3 middleboxes as well as VNFs that are transport layer endpoints; (ii) seamlessly supports multiple network stacks in the backend; and (iii) enables distributed implementation of VNFs via functions for distributed state and replica management. We have implemented a variety of VNFs using our library to demonstrate the expressiveness of our API. Our evaluation shows that building VNFs using libVNF can reduce the number of lines of code in the VNF by up to 50%. Further, optimizations in our library ensure that the performance of VNFs built with our library scales well with increasing number of CPU cores and distributed replicas.

P4 is a domain specific language to configure packet processing pipelines in programmable dataplane switches, and is a powerful idea towards realizing the goal of flexible software-defined networks. This paper presents pcube, a framework that provides a set of primitives to simplify the development of P4-based dataplane applications. pcube provides primitives for loops, summations, and other common operations on indexed state variables, which can be embedded within P4 code and unrolled by the pcube preprocessor. pcube also provides primitives to synchronize state variables across switches in distributed dataplane applications, which are automatically translated into P4 code to send and receive synchronization messages across multiple switches by pcube. We build example dataplane applications such as a distributed load balancer in our framework, and show that using pcube reduces the programming effort (in term of lines of code) significantly-by a factor of up to 5.4x.

Offloading computation to local controllers (closer to switches) has been a popular approach to designing scalable SDN controllers. We observe that, in addition to the offload of local switch-specific state, a subset of global state can also be offloaded to, and accessed at local controllers with suitable synchronization. We present the design and implementation of Cuttlefish, an SDN controller framework that adaptively offloads a portion of the application state (and computation) to local controllers. Cuttlefish uses developer-specified input to identify control messages that can be correctly processed at local controllers, and makes offloading decisions based on the cost of synchronizing the offloaded state across controllers. SDN applications use the Cuttlefish API to access the offloaded state, and Cuttlefish transparently manages the state synchronization, and redirection of control messages to the appropriate (central or local) controller. We have implemented Cuttlefish using the Floodlight SDN controller. Our evaluation shows that Cuttlefish applications achieve ~2X higher control plane throughput and ~50% lower control plane latency as compared to the traditional SDN design.

In recent years, WiFi has grown in capacity as well as deployment demand. WiFi system administrators (sysads) want a simple answer to the question “Is my WiFi network healthy?”, and a possible follow-up “What is wrong with it?”, if it is reported as “unhealthy”. But we are far from having such an interface today. It is this gap that this work attempts to fill. We present Witals, a system for WiFi performance diagnosis. We first design a causal diagnosis graph, that extensively identifies the underlying cause(s) of WiFi performance problems. Next, we identify a set of metrics corresponding to nodes in this causal graph. These metrics are measured in real-time by an operational AP, and help in quantifying the effect of each cause. We design a diagnosis algorithm based on the causal graph and the metrics, which ultimately presents a sanitized view of WiFi network health to the sysad. We have implemented a prototype of Witals on an enterprise grade 802.11n AP platform. Using a variety of controlled as well as real-life measurements, we show that our diagnosis framework follows ground truth accurately. Witals has also helped the sysads uncover some unexpected diagnoses.

Towards improving SDN control plane scalability, past work has proposed SDN controller frameworks that offload computation which depends on local state to controllers residing on the switches. Our work identifies another type of computation that can be offloaded to local controllers: that which depends on state that is generated globally but can be used within local controllers with loose synchronization. Because using such state locally incurs a synchronization cost, such offload makes sense only when the benefits of the offload out-weigh the synchronization cost. We present the design and implementation of Devolve-Redeem, an SDN controller framework that can offload computation to local controllers depending on the mix of various control messages in the incoming traffic. The offload decision in our framework is made by computing a cost metric that captures the relative costs of processing every control message at the central and local controllers, taking into account synchronization costs. The SDN application developer using our framework writes a single application that runs at both the central and local controllers, using our state management API to access offloadable state. Our framework migrates between various offload modes using the computed cost metric, by manipulating the rules in the SDN switches that forward control messages to the controllers. Our framework also transparently handles state synchronization between central and local controllers in a manner that is consistent with the offload mode. We have implemented the SDN-based LTE EPC application in our framework, and experiments with our prototype demonstrate the effectiveness of our adaptive offload framework.

How does a dense WiFi network perform, specifically for the common case of TCP download? While the empirical answer to this question is `poor', analysis and experimentation in prior work has indicated that TCP clocks itself quite well, avoiding contention-driven WiFi overload in dense settings. This paper focuses on measurements from a real-life use of WiFi in a dense scenario: a classroom where several students use the network to download quizzes and instruction material. We find that the TCP download performance is poor, contrary to that suggested by prior work. Through careful analysis, we explain the complex interaction of various phenomena which leads to this poor performance. Specifically, we observe that a small amount of upload traffic generated when downloading data upsets the TCP clocking, and increases contention on the channel. Further, contention losses lead to a vicious cycle of poor interaction with autorate adaptation and TCP's timeout mechanism. To reduce channel contention and improve performance, we propose a modification to the AP scheduling policy to improve the performance of large TCP downloads. Our solution, WiFiRR, picks only a subset of clients to be served by the AP during any instant, and varies this set of “active” clients periodically in a round-robin fashion over all clients to ensure that no client starves. We have done extensive evaluation of WiFiRR in simulation and in real settings. By reducing the number of contending nodes at any point of time, WiFiRR improves the download time of large TCP flows upto 3.5x of our classroom scenario. We also compare WiFiRR with state-of-the-art prior work WiFox, WiFiRR improves download time by 2.25× over WiFox.

This paper describes a system for improving user experience when accessing overloaded web servers. While several techniques exist today to build high-capacity web servers, little attention is paid to the fact that servers often crash when faced with transient overload, causing user experience to degrade sharply when incoming load exceeds capacity. Existing overload control mechanisms focus on some form of admission control to protect the web server from overload. However, all such techniques result in user requests failing, either due to timing out or receiving a "service unavailable" message. More importantly, there is no feedback to the frustrated user about when to retry again, leading to adhoc retries. This paper describes WebQ, a system consisting of two web proxies, that together simulate a virtual queue of web requests, and shape incoming load to match server capacity. Users accessing a server protected by WebQ receive a HTTP redirect response specifying a wait time in the virtual queue, and are automatically redirected to the web server upon expiration of the wait time. The wait times are calculated using an estimate of the server's capacity that is computed by WebQ. Users in the virtual queue are provided with a secure cryptographic token, which is checked to guarantee that the user has waited his prescribed time in the queue. We evaluate the ideas of WebQ using a real prototype implementation. Our experiments show that, with WebQ in place, users experience zero server failures and significantly better response times from a web server, even when the peak load is several times the provisioned capacity.

We present Witals, a system for WiFi performance diagnosis. In Witals, a live access point diagnoses the health of the WiFi network by examining the health of the air around it. Such diagnosis is critical functionality for sysads, and is an important addition to the state of the art. In the demo we will show WiFi performance diagnosis by Witals live at MobiCom venue and also from past collected traces in other settings & for controlled experiments.

Over the past few years, researchers have developed many cross-layer wireless protocols to improve the performance of wireless networks. Experimental evaluations of these protocols have been carried out mostly using software-defined radios, which are typically two to three orders of magnitude slower than commodity hardware. FPGA-based platforms provide much better speeds but are quite difficult to modify because of the way high-speed designs are typically implemented. Experimenting with cross-layer protocols requires a flexible way to convey information beyond the data itself from lower to higher layers, and a way for higher layers to configure lower layers dynamically and within some latency bounds. One also needs to be able to modify a layer's processing pipeline without triggering a cascade of changes. We have developed Airblue, an FPGA-based software radio platform, that has all these properties and runs at speeds comparable to commodity hardware. We discuss the design philosophy underlying Airblue that makes it relatively easy to modify it, and present early experimental results.

This article presents the design, implementation, analysis, and experimental evaluation of speak-up , a defense against application-level distributed denial-of-service (DDoS), in which attackers cripple a server by sending legitimate-looking requests that consume computational resources (e.g., CPU cycles, disk). With speak-up, a victimized server encourages all clients, resources permitting, to automatically send higher volumes of traffic . We suppose that attackers are already using most of their upload bandwidth so cannot react to the encouragement. Good clients, however, have spare upload bandwidth so can react to the encouragement with drastically higher volumes of traffic. The intended outcome of this traffic inflation is that the good clients crowd out the bad ones, thereby capturing a much larger fraction of the server's resources than before. We experiment under various conditions and find that speak-up causes the server to spend resources on a group of clients in rough proportion to their aggregate upload bandwidths, which is the intended result.

This paper presents SoftRate, a wireless bit rate adaptation protocol that is responsive to rapidly varying channel conditions. Unlike previous work that uses either frame receptions or signal-to-noise ratio (SNR) estimates to select bit rates, SoftRate uses confidence information calculated by the physical layer and exported to higher layers via the SoftPHY interface to estimate the prevailing channel bit error rate (BER). Senders use this BER estimate, calculated over each received packet (even when the packet has no bit errors), to pick good bit rates. SoftRate's novel BER computation works across different wireless environments and hardware without requiring any retraining. SoftRate also uses abrupt changes in the BER estimate to identify interference, enabling it to reduce the bit rate only in response to channel errors caused by attenuation or fading. Our experiments conducted using a software radio prototype show that SoftRate achieves 2X higher throughput than popular frame-level protocols such as SampleRate and RRAA. It also achieves 20% more throughput than an SNR-based protocol trained on the operating environment, and up to 4X higher throughput than an untrained SNR-based protocol. The throughput gains using SoftRate stem from its ability to react to channel variations within a single packet-time and its robustness to collision losses.

Network intrusion detection and prevention systems are vulnerable to evasion by attackers who craft ambiguous traffic to breach the defense of such systems. A normalizer is an inline network element that thwarts evasion attempts by removing ambiguities in network traffic. A particularly challenging step in normalization is the sound detection of inconsistent TCP retransmissions, wherein an attacker sends TCP segments with different payloads for the same sequence number space to present a network monitor with ambiguous analysis. Normalizers that buffer all unacknowledged data to verify the consistency of subsequent retransmissions consume inordinate amounts of memory on highspeed links. On the other hand, normalizers that buffer only the hashes of unacknowledged segments cannot verify the consistency of 20-30% of retransmissions that, according to our traces, do not align with the original transmissions. This paper presents the design of RoboNorm, a normalizer that buffers only the hashes of unacknowledged segments, and yet can detect all inconsistent retransmissions in any TCP byte stream. RoboNorm consumes 1-2 orders of magnitude less memory than normalizers that buffers all unacknowledged data, and is amenable to a high-speed implementation. RoboNorm is also robust to attacks that attempt to compromise its operation or exhaust its resources.

The Border Gateway Protocol (BGP), the current inter domain routing protocol in the Internet, has two modes of operation: eBGP (External BGP), used to exchange routing information between autonomous systems, and iBGP (Internal BGP), used to propagate that information within an autonomous system (AS). This paper focuses on the construction of an iBGP session configuration that guarantees two correctness properties - loop-free forwarding paths and complete visibility to all eBGP-learned best routes - while attempting to minimize the number of iBGP sessions (for scalability) and ensuring that the constructed configuration guarantees the two correctness properties even in the face of link failures and IGPpath changes. Our algorithm constructs an iBGP configuration based on route reflectors, a commonly used way to control the number of iBGP sessions. The algorithm, BGPSep, uses the notion of a graph separator, a (small) set of nodes that partition a graph into connected components of roughly equal sizes, recursively applies this idea to the connected components, and produces a route reflector hierarchy and the associated iBGP sessions. We prove thatBGPSep guarantees the desired correctness properties, andevaluate an implementation of the BGPSep algorithm on several real-world and simulated network topologies. Across these topologies, we find that the number of iBGP sessions with is afactor of 2.5 to 5 times smaller than with a \"full mesh\" iBGP, while guaranteeing the desired correctness properties.

This paper presents the design, implementation, analysis, and experimental evaluation of speak-up, a defense against application-level distributed denial-of-service (DDoS), in which attackers cripple a server by sending legitimate-looking requests that consume computational resources (e.g., CPU cycles, disk). With speak-up, a victimized server encourages all clients, resources permitting, to automatically send higher volumes of traffic. We suppose that attackers are already using most of their upload bandwidth so cannot react to the encouragement. Good clients, however, have spare upload bandwidth and will react to the encouragement with drastically higher volumes of traffic. The intended outcome of this traffic inflation is that the good clients crowd out the bad ones, thereby capturing a much larger fraction of the server's resources than before. We experiment under various conditions and find that speak-up causes the server to spend resources on a group of clients in rough proportion to their aggregate upload bandwidth. This result makes the defense viable and effective for a class of real attacks.