Practitioners of a growing number of scientific and artificial-intelligence (AI) applications use High-Performance Wide-Area Networks (HP-WANs) for moving massive data sets between remote facilities. Accurate prediction of the flow completion time (FCT) is essential in these data-transfer workflows because compute and storage resources are tightly scheduled and expensive. We assess the viability of three TCP congestion control algorithms (CUBIC, BBRv1, and BBRv3) for massive data transfers over public HP-WANs, where limited control of critical data-path parameters precludes the use of Remote Direct Memory Access (RDMA) over Converged Ethernet (RoCEv2), which is known to outperform TCP in private HP-WANs. Extensive experiments on the FABRIC testbed indicate that the configuration control limitations can also hinder TCP, especially through microburst-induced packet losses. Under these challenging conditions, we show that the highest FCT predictability is achieved by combination of BBRv1 with the application of traffic shaping before the HP-WAN entry points.

Use cases for network determinism may involve the traversal of multiple network domains. A typical example is the edge-cloud virtualization of the controllers for industrial automation, where the end-to-end data path may include a private mobile network and a wired local-area network within the factory premises, a public transport network between the factory and the edge-cloud data center, and a switching fabric within the same data center. Maintaining deterministic guarantees throughout these domains, likely operated by different providers, is a daunting task when each domain relies on its own control and data-plane technologies for satisfaction of its allocated portion of the end-to-end requirements. We propose a universal model for determinism in wired packet networks that simplifies the realization of deterministic services over chains of network domains. The model builds on a novel traffic shaper where routing information constrains the packet scheduling decisions. The shaper, and therefore the provisioning of resources for every deterministic service, is only needed at the ingress edge of each wired domain. We conjecture perdomain latency bounds enforced by the shaper and corroborate them with experimental evidence. The bounds are easy to combine for multi-domain network paths.

5G will transform the cellular network from a generic supplier of high-speed data to a flexible, super-fast, and highly-reliable communications medium that supports diverse sets of services and applications. Low packet latency is a critical enabler for the interactivity features of many of these applications, but maybe compromised in saturated packets buffers if the buffers are oversized, as is often the case in the RLC layer of the 3GPP RAN stack. To protect all interactive applications from excessive buffering, we completely revise the flow-control component of a prior solution of ours that splits the per-bearer buffer between the RLC and PDCP layers and regulates the packet flow between the two buffers so that queue build-up in the RLC layer is constrained. On a cellular link shared with a greedy TCP flow, our new enhanced bearer buffer (EBB) design with continuous flow control (CFC) reduces the latency of packets from low-bandwidth interactive applications by at least 80% compared to previous EBB versions, and by orders of magnitude compared to the default bearer configuration with a large RLC buffer. The CFC also benefits throughput-intensive interactive applications like augmented and virtual reality because it confines the buffering within the PDCP layer, where it can be effectively controlled with active queue management techniques.

5G is expected to expand the set of interactive applications enabled by the mobile network. Low end-to-end packet latency is a fundamental requirement for proper operation of those applications, but may be compromised in packet buffers shared with elastic applications that use greedy TCP sources for end-to-end transport. The accumulation of queuing delay by the elastic application degrades the latency for the interactive application, however light the throughput of the latter may be (as is the case with online gaming and over-the top voice). Buffer sharing is unavoidable in the RLC layer of the 3GPP RAN stack. To minimize its negative effect on interactive applications, we split the buffering between the RLC and PDCP layers. Then we equip the PDCP buffer with per-flow queues, and apply to the RLC buffer a new dynamic sizing mechanism that enforces the lowest queuing delay that is compatible with the existing configuration of the RLC connection. On a cellular link shared with a greedy TCP flow, our dynamic sizing solution can reduce the queuing delay of PING packets by up to two orders of magnitude compared to the default configuration with an RLC-only buffer of fixed size.

Applications are typically offered a single, generic option for end-to-end transport, which most commonly consists of the particular flavor of TCP that runs in the server host. This arrangement does not suit well the diversity of the requirements that individual applications pose on network metrics like throughput and delay. Link-Coupled TCP (LCTCP) is a new transport solution that leverages the architectural trends of 5G networks to enable accurate satisfaction of the unique requirements of each application. LCTCP first isolates the 5G access link as the only possible network bottleneck for the application flow, then establishes a lightweight signaling channel between the link buffer and the application server to convey critical information for flexible control of the data source. LCTCP can be deployed in the network without modification of the TCP clients. We use a Linux prototype to demonstrate its feasibility and effectiveness.

The IEEE 802.11 standards define a distributed scheme for Wi-Fi access points and stations to fairly share the wireless medium. Even if fully standard-compliant, Wi-Fi devices from different vendors have implementation differences that lead to disparities in their ability to access the medium. Erratic upper-layer behaviors become manifest when devices that exhibit such disparities inter-operate within one network. In this paper we show examples of those behaviors based on common use cases. We find that a primary cause of performance inconsistency for most network applications is the uneven ability of different IEEE 802.11 devices to access the shared medium for transmission of TCP acknowledgments, further aggravated by the excessive size of the buffers where those packets are queued before transmission. We devise and validate in a real network an effective solution for Linux hosts that is based on the link-layer priorities of IEEE 802.11e. This solution allows researchers conducting Wi-Fi experiments to collect device-independent results, and application providers to guarantee a consistent experience to their users across different devices.

In a congested network link, synchronization effects between bandwidth-sharing TCP flows cause wide queue length oscillations, which may translate into poor link utilization if insufficiently buffered. We introduce global synchronization protection (GSP), a simple extension to the ordinary operation of a tail-drop queue that safely suppresses the flow synchronization. Our minimalistic solution is well suited for scaling with leading-edge link rates: it adds only few extra operations in the fast path and does not require accelerated memory access compared to the line rate. GSP makes it easier to provide advanced control of TCP congestion in high-speed links and in low-power packet processing hardware. Using experiments with a Linux prototype of GSP we show that, despite its exclusive focus on removing global synchronization, the new scheme performs as well as far more complex active queue management (AQM) schemes like CoDel and PIE.

Network emulation strikes the balance between using real machines on full-fledged networks and running software models of applications and networks in simulation environments. Advanced Linux features make it possible to emulate entire networks on a single machine, enabling experiments that are much easier to run and repeat. However, some of these features were not designed with the primary purpose of supporting emulation frameworks: if not carefully configured, they can lead to erroneous results. In this paper we note a few properties of Linux-based network emulation that researchers should consider while planning their experiments. We also recommend a particular configuration that improves the fidelity of emulation results.

We tested AQM techniques on their ability to influence end-user QoE, especially HTTP Adaptive Streaming (HAS) based video traffic on fixed access networks. Using ns-2 we built a set of simulation scenarios for realistic Internet traffic. Based on data from recent Internet measurements, there are three major types of traffic flowing through the Internet: HTTP web traffic; HAS video traffic and progressive download (PD) video traffic. We modeled HTTP web traffic using Internet statistics published by Google. We generated dynamic HAS traffic by implementing a HAS client. We modeled PD video traffic using statistics from Youtube. We used published research to convert data from simulation traces into QoE metrics. We used realistic mixes of the three types of traffic with different loading conditions to test Random Early Detection (RED), CoDel and Tail-drop queuing on DSL access networks. We also tested various options in HAS for their impact on its QoE. Most AQM configurations improved the QoE metrics of HAS and web traffic. They improved fairness among HAS streams, avoided HAS underflow in a majority of the cases where it would occur with Tail-drop, and tangibly reduced the latency for web page downloads. Sensitivity analysis with RED parameters confirmed that the improved QoE of HAS and web traffic was relatively insensitive to the exact RED parameter set.

Rate adaptation is a family of technologies driven by the expectation that large energy savings can be achieved in packet networks by dynamically adjusting the capacity of network components to the load that they are required to sustain. In this paper we focus on packet-timescale rate adaptation (PTRA) techniques, which apply to individual traffic processing chips in the circuit packs of network systems. We look at the available options for PTRA implementation and compare their performance in realistic multi-device configurations. We find that in linear multi-device topologies the sleep-state-exploitation (SSE) scheme, which only adds a sleep state to the ordinary full-capacity state, offers the best compromise between energy savings and the unavoidable packet delay degradation of PTRA.

The scalability and energy efficiency of future network equipment will critically depend on the ability to confine the memories that implement the packet buffers within the same traffic management chips that process and forward the packets. Despite massive research efforts aimed at trimming its demand of large buffers for the accommodation of TCP traffic, the bandwidth-delay product (BDP) rule remains to-date the dominant criterion for the sizing of packet buffers in commercial network elements, and arguably the only cause for their implementation in off-chip memories. Only the lack of a valid alternative justifies the lasting popularity of conventional buffer management methods for TCP traffic such as Tail Drop and Random Early Detection (RED), which fail to reconcile small buffer sizes with high-end throughput performance. Our contribution is twofold. First, we show that the RED algorithm is intrinsically flawed because of the way it maps buffer occupancy levels onto packet drop probabilities. Second, we introduce Periodic Early Detection (PED), a buffer management scheme with touchless configuration that sustains 100% link utilization using only 2.5% of the memory required by the BDP rule. While a more comprehensive study of PED's properties is in order, the clear superiority of the scheme under common benchmarking setups places it at the forefront of the candidate enablers for the on-chip implementation of buffer memories.

To maximize the energy efficiency of packet networks, energy use in network equipment should scale rigorously with the traffic load. Rate adaptation is gaining popularity as a promising framework for achieving energy proportionality in the data-path components of routers and switches, but a thorough characterization of its energy-saving capabilities and impact on network performance is still lacking. In this paper, we present novel rate-adaptation schemes that are easy to implement, amenable to incremental deployment in packet switches and routers, and whose negative effect on end-to-end packet delays is bounded. We derive tight lower and upper bounds on energy consumption that we use to further refine the design of our schemes. Our analysis indicates that large-scale deployments of rate adaptation can enable substantial energy savings, as large gains can be attained in individual data-path devices under parametric configurations that do not compromise the end-to-end delay performance of the network.

Due to the changed economic environment, the rush to implementing packet switches with switching capacities above 1 Tb/s, which had proceeded at a frantic pace for some years, has slowed down considerably. Most service providers do not foresee the deployment of switches and routers with gigantic capacities in the near future. The immediate interest does now rarely go beyond the subterabit range, with a sweet spot between 120-640 Gb/s, where the emphasis is on feature-rich systems that enable the convergence of legacy services with new emerging data services. Although the current focus is on smaller switches, it is still relevant to understand their evolution path to multiterabit capacities. The scalability issues are also critical to reduce complexity and simplify implementation, in order to push the limits of what can be achieved in the switches within current economic and market constraints. We analyze the current state of the art of practical large packet switches and routers, and discuss the issues affecting their scalability. Our approach is pragmatic, with most of our attention devoted to three major scalability aspects: implementation, support of quality of service, and multicasting. After a general discussion of these issues, we show their impact on the most popular switch architectures.

In small cell-based Internet protocol routers, multicast traffic is generally handled by appending to each cell a local multicast label (LML) containing a bitmap with as many bits as switch ports, so as to identify the ports to which a copy of the cell has to be transferred. This approach is not feasible for switches having 128 ports or more, because the LML length would rise above 16 bytes, thus representing an intolerable overhead, given the small size of cells (typically 64 bytes). We discuss both static and adaptive lossy compression algorithms to reduce the size of LMLs to be attached to multicast cells, at the price of the delivery of cells to a larger set of outputs than necessary, and we compare the compression algorithms performance in terms of switch bandwidth waste, using both analytical and simulation models.

We present the Distributed Frame-Definition Algorithm (DFDA), a novel scheduling policy for input-queued switches with virtual output queueing at the input line cards. The DFDA effectively supports the integration of traffic classes with diverse Quality-of-Service (QoS) requirements (as compelled by emerging QoS frameworks such as Differentiated Services), and scales well with the aggregate capacity of the switch because of the limited complexity of its distributed implementation. The input and output tine cards exchange a minimal amount of control information to define the periodic service schedule that regulates transit through the switch fabric. The composition of the service schedule dynamically adapts to the tight bandwidth requirements of real-time traffic and to the load fluctuations of best-effort traffic. We assess the performance of the DFDA through simulation, and compare it with the most popular scheduling algorithms for input-queued switches. Our experiments show that the DFDA generally sustains higher throughput than the schemes of the prior art.

We investigate techniques for the support of multicast traffic in IP routers that have a large number of switch fabric ports (from 128 to 1024) and internally operate on relatively small fixed-size data units (cells of 64 bytes each). In small packet switches, multicast traffic is typically handled by prepending a local multicast label (LML) to each cell. The LML consists of a bitmap with as many bits as switch ports. The bitmap identifies the set of ports to which copies of the cell must be transferred. The bitmap approach is no longer feasible in switches with 128 ports or more, where the LML length cannot be smaller than 16 bytes, an intolerable overhead when the internal cell payload is only 64 bytes. We devise several compression algorithms, both static and adaptive, to reduce the size of the LML's to be attached to multicast cells. The algorithms define compressed representations of the distribution sets of the multicast cells, trading the additional bandwidth needed to transfer redundant cell copies (i.e., copies directed to outputs that do not belong to the actual distribution sets) for the cell header overhead otherwise needed to deliver the cells only to the proper outputs. We use simulation experiments to compare the performance of the compression algorithms.

Multistage packet switches that feature a limited amount of buffers in the switching fabric and distribute most of their buffering capacity over the port cards have recently gained popularity due to their scalability properties and flexibility in supporting quality-of-service (QoS) guarantees. In such switches, the replication of multicast packets typically occurs in-the fabric. This approach minimizes the amount of resources needed to sustain the internal expansion in traffic volume due to multicasting, but also exposes multicast flows to head-of-line (HOL) blocking in the ingress port cards. The distributed scheduler that arbitrates the transfer of packets through the switch and the flow-control scheme that restricts access to the fabric buffers have the most critical role in safeguarding the QoS guarantees of multicast flows against HOL blocking. We add minimal overhead to a well-known QoS framework for multistage packet switches to define the generalized distributed multilayered scheduler (G-DMS), which achieves full support of QoS guarantees for both unicast and multicast flows. The main novelty of the G-DMS is in the mechanism that regulates access to the fabric buffers, which combines selective backpressure with the capability of dropping copies of multicast packets that violate the negotiated profiles of the corresponding flows.

Weighted fair queueing (WFQ) schedulers and shapers have gained wide popularity in current routers and switches because of their advanced quality-of-service features and very low implementation cost. A well-known, intrinsic limitation of WFQ schedulers is the coupling between the bandwidth and delay guarantees that they can provide. Because of this coupling, the support of traffic with low bandwidth and stringent delay requirements, most notably voice, is always problematic with WFQ schedulers. We present the decoupled fair queueing (DFQ) server, which is specifically designed for the support of voice traffic in packet-switched networks. The DFQ server maintains the very low implementation complexity of WFQ schedulers, yet it guarantees low delay to flows requesting relatively low bandwidth. This is achieved by judiciously over-allocating the scheduling rates, and by combining the scheduler with a policer, so as to guarantee that the voice flows only receive the desired bandwidth and do not damage one another, nor the other types of flows that are handled by the server.

Multiple-module, multistage packet switching systems are gaining popularity as a scalable solution to the ever increasing demand for the aggregate switching capacity. Due to additional contention points between the stages, such systems differ in their behavior from single module output buffered switches, which serve as a model for most of the advanced quality of service (QoS) scheduling algorithms. Recently, a distributed scheduling reference architecture has been proposed to extend the QoS provisioning framework to the practical multistage switches. While keeping all the per-flow state information in the port card, it aggregates the individual traffic flows into a few QoS based fabric channels and relies on the intelligent selective feedback to ensure that the QoS guarantees of the individual flows are met. In this paper we discuss a novel credit-based feedback mechanism that allows to aggregate multiple traffic components with diverse QoS requirements into the same downstream FIFO queue. We specifically apply it to merge the guaranteed-bandwidth (GB) and best-effort (BE) QoS channels of the distributed scheduling architecture into a single non-guaranteed-delay queue in the switch fabric. The mechanism allows to satisfy the bandwidth requirements of the GB traffic while maximizing the throughput of the BE traffic and distributes the available excess bandwidth between different types of traffic fairly. The presentation is supported by fluid stationary analysis and packet-level simulations.

The continuous growth in the demand for diversified quality-of-service (QoS) guarantees in broadband networks introduces new challenges in the design of packet switches that scale to large switching capacities. Packet scheduling is the most critical function involved in the provision of individual bandwidth and delay guarantees to the switched flows. Most of the scheduling techniques proposed so far assume the presence in the switch of a single contention point, residing in front of the outgoing links. Such an assumption is not consistent with the highly distributed nature of many popular architectures for scalable switches, which typically have multiple contention points, located in both ingress and egress port cards, as well as in the switching fabric. We define a distributed multilayered scheduler (DMS) to provide differentiated QoS guarantees to individual end-to-end flows in packet switches with multiple contention points. Our scheduling architecture is simple to implement, since it keeps per-flow scheduling confined within the port cards, and is suitable to support guaranteed and best-effort traffic in a wide range of QoS frameworks in both IP and ATM networks.

The total implementation cost of schedulers which approximate the generalized processor sharing (GPS) policy is dominated by the complexity of maintaining and sorting the time-stamps for all connections. Several approaches have been proposed which reduce the cost of the sorting operation and only introduce a small degradation in the delay bounds of the scheduler; they include logarithmic calendar queues and schedulers supporting a discrete set of guaranteed rates. All these techniques still require computing and storing one time-stamp per connection, thus maintaining the cost of GPS-related algorithms clearly higher than that of less sophisticated schedulers. Furthermore, in the case of the discrete-rate approach, the complexity increases linearly with the number of supported rates, thus making it attractive only for relatively small numbers of rates. In this paper, we introduce a discrete-rate GPS-related scheduler which does not require the computation and storage of one time-stamp per connection, and only maintains a single time-stamp per rate. The elimination of the per-connection time-stamps has no negative effect on the delay bounds. Then, we present a generalized discrete-rate approach, which uses a given number of FIFO queues to support a larger number of guaranteed rates, and only introduces a modest degradation in delay bounds for certain rates. The technique can be applied to our no-per-connection-time-stamp scheduler, as well as to any discrete-rate scheduler.

The minimum-delay self-clocked fair queueing (MD-SCFQ) algorithm that we present in this paper is the first algorithm that achieves the same delay bounds as the packet by packet version of generalized processor sharing (P-GPS), has fairness properties similar to P-GPS, and uses a system-potential function of 0(1) complexity. To prove that MD-SCFQ achieves the same delay bounds of P-GPS, we have shown that the fluid version of MD-SCFQ belongs to the class of rate proportional server (RPS) schedulers; thus, its packet-by-packet version inherits all the well-known single-node and multiple-node delay properties of P-RPS schedulers. We have also computed the fairness index of MD-SCFQ, and show that is very close to that of P-GPS. Because of the properties of its system potential, MD-SCFQ, when combined with a shaper which declares packets eligible for scheduling according to their virtual starting times, is a work-conserving worst-case fair scheduler.