The three decade struggle to ensure Internet data confidentiality---a key aspect of communications privacy---is finally behind us. Encryption is fast, secure, and standard in all browsers, modern transports, and major protocols. Yet it has long seemed that network privacy is not unified by core principles but a grab bag of techniques and ideas applied to an equally wide range of applications, contexts, layers of infrastructure, and software stacks.

We present our experience with QUIC, an encrypted, multiplexed, and low-latency transport protocol designed from the ground up to improve transport performance for HTTPS traffic and to enable rapid deployment and continued evolution of transport mechanisms. QUIC has been globally deployed at Google on thousands of servers and is used to serve traffic to a range of clients including a widely-used web browser (Chrome) and a popular mobile video streaming app (YouTube). We estimate that 7% of Internet traffic is now QUIC. We describe our motivations for developing a new transport, the principles that guided our design, the Internet-scale process that we used to perform iterative experiments on QUIC, performance improvements seen by our various services, and our experience deploying QUIC globally. We also share lessons about transport design and the Internet ecosystem that we learned from our deployment.

TCP congestion control algorithms implicitly assume that the per-flow throughput is at least a few packets per round trip time. Environments where this assumption does not hold, which we refer to as small packet regimes, are common in the contexts of wired and cellular networks in developing regions. In this paper we show that in small packet regimes TCP flows experience severe unfairness, high packet loss rates, and flow silences due to repetitive timeouts. We propose an approximate Markov model to describe TCP behavior in small packet regimes to characterize the TCP breakdown region that leads to repetitive timeout behavior. To enhance TCP performance in such regimes, we propose Timeout Aware Queuing (TAQ), a readily deployable in-network middlebox approach that uses a multi-level adaptive priority queuing algorithm to reduce the probability of timeouts, improve fairness and performance predictability. We demonstrate the effectiveness of TAQ across a spectrum of small packet regime network conditions using simulations, a prototype implementation, and testbed experiments.

The traditional problem of end-hosts efficiently and fairly utilizing end-to-end paths becomes significantly harder when the end-hosts are multihomed. Such is the case, for instance, when an end-host has simultaneous connectivity through several service providers, or when a mobile device is simultaneously connected via both a wireless LAN and a cellular network. A multihoming-aware transport protocol, such as MPTCP or SCTP, that sends data over the multiple resulting end-to-end paths must be fair to other flows in the network while being able to maximize its own throughput. In this paper, we present Dynamic Window Coupling (DWC), a multipath congestion control mechanism that seeks to achieve both these goals. DWC uses loss and delay signals to detect shared bottlenecks, explicitly grouping and sharing congestion control across subflows on paths that have a common bottleneck, while separating congestion control for subflows on paths with distinct bottlenecks. DWC detects shifting bottlenecks in the network and responds by dynamically regrouping subflows. Simulations demonstrate that DWC detects shared bottlenecks under most network topologies and conditions that we considered, regroups subflows correctly as bottlenecks shift, aggregates throughput across distinct bottlenecks, and is fair to other TCP flows at all bottlenecks.

Many network links in developing regions operate in the sub-packet regime, an environment where the typical per-flow throughput is less than 1 packet per round-trip time. TCP and other common congestion control protocols break down in the sub-packet regime, resulting in severe unfairness, high packet loss rates, and flow silences due to repetitive timeouts. To understand TCP's behavior in this regime, we propose a model particularly tailored to high packet loss-rates and relatively small congestion window sizes. We validate the model under a variety of network conditions.

In both TCP and SCTP, selectively acked (SACKed) out-of-order data is implicitly renegable; that is, the receiver can later discard SACKed data. The possibility of reneging forces the transport sender to maintain copies of SACKed data in the send buffer until they are cumulatively acked. In this paper, we investigate the situation where all out-of-order data is non-renegable, such as when the data has been delivered to the application, or when the receiver simply never reneges. Using simulations, we show that SACKs result in inevitable send buffer wastage, which increases as frequency of loss events and loss recovery durations increase. We introduce a fundamentally new ack mechanism, Non-Renegable Selective Acknowledgments (NR-SACKs), for SCTP. Using NR-SACKs, an SCTP receiver can explicitly identify some or all out-of-order data as being non-renegable, allowing the sender to free up send buffer sooner than if the data were only SACKed. Simulation comparisons show that NR-SACKs enable efficient utilization of a transport senderpsilas memory. Further investigations show that NR-SACKs also improve throughput in Concurrent Multipath Transfer (CMT) [4].

We propose using the Stream Control Transmission Protocol (SCTP), a recent IETF transport layer protocol, for reliable web transport. Although TCP has traditionally been used, we argue that SCTP better matches the needs of HTTP-based network applications. This position paper discusses SCTP features that address: (i) head-of-line blocking within a single TCP connection, (ii) vulnerability to network failures, and (iii) vulnerability to denial-of-service SYN attacks. We discuss our experience in modifying the Apache server and the Firefox browser to benefit from SCTP, and demonstrate our HTTP over SCTP design via simple experiments. We also discuss the benefits of using SCTP in other web domains through two example scenarios ? multiplexing user requests, and multiplexing resource access. Finally, we highlight several SCTP features that will be valuable to the design and implementation of current HTTP-based client-server applications.

Previously, we studied the performance of concurrent multipath transfer using SCTP multihoming (CMT) under the assumption of an infinite receive buffer (rbuf). Here, we study CMT performance when a sender is constrained by the rbuf. We demonstrate using simulation that if two paths are used for CMT, the lower quality (i.e., higher loss rate) path degrades overall throughput of an rbuf-constrained CMT association by blocking the rbuf. We demonstrate that a wise retransmission policy can alleviate some of the throughput degradation by reducing the rbuf blocking problem. We present and discuss CMT performance using several retransmission policies and constrained rbuf values of 16 KB, 32 KB, 64 KB, 128 KB, and 256 KB. While rbuf blocking cannot be eliminated, it can be reduced by choice of retransmission policy - a facility available to only the transport layer.

In recent years several enhancements to TCP congestion control and loss recovery mechanisms have been proposed and accepted as Internet standards. In this paper, we examine these enhancements in detail. We consider five TCP enhancements: (1) selective acknowledgements (SACK) and the SACK-based loss recovery algorithm; (2) increasing initial congestion window; (3) limited transmit; (4) appropriate byte counting; and (5) early retransmit. We present results from active measurements performed on web servers on the state of deployment of these enhancements. Our results show that while several web servers support TCP enhancements, the majority still use previous standards for congestion control and loss recovery. Using simulation, we quantify the cumulative effect of these TCP enhancements on web based and bulk data transfers. We hope that such an evaluation provides a clearer view of the applicability of these enhancements, and further motivation for their implementation.