Programmable ASIC technology enables the switching data plane to rapidly support emergent technologies such as VNF offloading, custom tunneling and in-band telemetry. We propose a new approach for a "hybrid mode" of ASIC programmability, which maintains a discrete legacy hardware pipeline and control functions (e.g. routing, bridging) while providing a way to extend it. This places requirements on the switching hardware, programming language, data plane APIs and the network OS in order to achieve this goal. In this paper we present two hardware agnostic hybrid mode applications using a Mellanox programmable switch ASIC, P4-16 programming language, SAI flexible APIs and the SONIC Open Network OS and Linux TC. Also applications based on the Onyx OS and Spectrum SDK is discussed as a hardware specific example.

Network reliability is critical for large clouds and online service providers like Microsoft. Our network is large, heterogeneous, complex and undergoes constant churns. In such an environment even small issues triggered by device failures, buggy device software, configuration errors, unproven management tools and unavoidable human errors can quickly cause large outages. A promising way to minimize such network outages is to proactively validate all network operations in a high-fidelity network emulator, before they are carried out in production. To this end, we present CrystalNet, a cloud-scale, high-fidelity network emulator. It runs real network device firmwares in a network of containers and virtual machines, loaded with production configurations. Network engineers can use the same management tools and methods to interact with the emulated network as they do with a production network. CrystalNet can handle heterogeneous device firmwares and can scale to emulate thousands of network devices in a matter of minutes. To reduce resource consumption, it carefully selects a boundary of emulations, while ensuring correctness of propagation of network changes. Microsoft's network engineers use CrystalNet on a daily basis to test planned network operations. Our experience shows that CrystalNet enables operators to detect many issues that could trigger significant outages.

Debugging faults in complex networks often requires capturing and analyzing traffic at the packet level. In this task, datacenter networks (DCNs) present unique challenges with their scale, traffic volume, and diversity of faults. To troubleshoot faults in a timely manner, DCN administrators must a) identify affected packets inside large volume of traffic; b) track them across multiple network components; c) analyze traffic traces for fault patterns; and d) test or confirm potential causes. To our knowledge, no tool today can achieve both the specificity and scale required for this task.

Load balancing is a foundational function of datacenter infrastructures and is critical to the performance of online services hosted in datacenters. As the demand for cloud services grows, expensive and hard-to-scale dedicated hardware load balancers are being replaced with software load balancers that scale using a distributed data plane that runs on commodity servers. Software load balancers offer low cost, high availability and high flexibility, but suffer high latency and low capacity per load balancer, making them less than ideal for applications that demand either high throughput, or low latency or both. In this paper, we present Duet, which offers all the benefits of software load balancer, along with low latency and high availability -- at next to no cost. We do this by exploiting a hitherto overlooked resource in the data center networks -- the switches themselves. We show how to embed the load balancing functionality into existing hardware switches, thereby achieving organic scalability at no extra cost. For flexibility and high availability, Duet seamlessly integrates the switch-based load balancer with a small deployment of software load balancer. We enumerate and solve several architectural and algorithmic challenges involved in building such a hybrid load balancer. We evaluate Duet using a prototype implementation, as well as extensive simulations driven by traces from our production data centers. Our evaluation shows that Duet provides 10x more capacity than a software load balancer, at a fraction of a cost, while reducing latency by a factor of 10 or more, and is able to quickly adapt to network dynamics including failures.

Clos-based networks including Fat-tree and VL2 are being built in data centers, but existing per-flow based routing causes low network utilization and long latency tail. In this paper, by studying the structural properties of Fat-tree and VL2, we propose a per-packet round-robin based routing algorithm called Digit-Reversal Bouncing (DRB). DRB achieves perfect packet interleaving. Our analysis and simulations show that, compared with random-based load-balancing algorithms, DRB results in smaller and bounded queues even when traffic load approaches 100%, and it uses smaller re-sequencing buffer for absorbing out-of-order packet arrivals. Our implementation demonstrates that our design can be readily implemented with commodity switches. Experiments on our testbed, a Fat-tree with 54 servers, confirm our analysis and simulations, and further show that our design handles network failures in 1-2 seconds and has the desirable graceful performance degradation property.

Reliable Group Data Delivery (RGDD) is a pervasive traffic pattern in data centers. In an RGDD group, a sender needs to reliably deliver a copy of data to all the receivers. Existing solutions either do not scale due to the large number of RGDD groups (e.g., IP multicast) or cannot efficiently use network bandwidth (e.g., end-host overlays). Motivated by recent advances on data center network topology designs (multiple edge-disjoint Steiner trees for RGDD) and innovations on network devices (practical in-network packet caching), we propose Datacast for RGDD. Datacast explores two design spaces: 1) Datacast uses multiple edge-disjoint Steiner trees for data delivery acceleration. 2) Datacast leverages in-network packet caching and introduces a simple soft-state based congestion control algorithm to address the scalability and efficiency issues of RGDD. Our analysis reveals that Datacast congestion control works well with small cache sizes (e.g., 125KB) and causes few duplicate data transmissions (e.g., 1.19%). Both simulations and experiments confirm our theoretical analysis. We also use experiments to compare the performance of Datacast and BitTorrent. In a BCube(4, 1) with 1Gbps links, we use both Datacast and BitTorrent to transmit 4GB data. The link stress of Datacast is 1.01, while it is 1.39 for BitTorrent. By using two Steiner trees, Datacast finishes the transmission in 16.9s, while BitTorrent uses 52s.

Reliable Group Data Delivery (RGDD) is a pervasive traffic pattern in data centers. In an RGDD group, a sender needs to reliably deliver a copy of data to all the receivers. Existing solutions either do not scale due to the large number of RGDD groups (e.g., IP multicast) or cannot efficiently use network bandwidth (e.g., end-host overlays).

There have been some serious concerns about the TCP performance in data center networks, including the long completion time of short TCP flows in competition with long TCP flows, and the congestion due to TCP incast. In this paper, we show that a properly tuned instant queue length based Explicit Congestion Notification (ECN) at the intermediate switches can alleviate both problems. Compared with previous work, our approach is appealing as it can be supported on current commodity switches with a simple parameter setting and it does not need any modification on ECN protocol at the end servers. Furthermore, we have observed a dilemma in which a higher ECN threshold leads to higher throughput for long flows whereas a lower threshold leads to more senders on incast under buffer pressure. We address this problem with a switch modification only scheme - dequeue marking, for further tuning the instant queue length based ECN to achieve optimal incast performance and long flow throughput with a single threshold value. Our experimental study demonstrates that dequeue marking is effective for increasing the maximum incast senders close to the performance limit of ECN, achieving a gain anywhere from 16% to 140%.

In this paper, we propose virtual data center (VDC) as the unit of resource allocation for multiple tenants in the cloud. VDCs are more desirable than physical data centers because the resources allocated to VDCs can be rapidly adjusted as tenants' needs change. To enable the VDC abstraction, we design a data center network virtualization architecture called SecondNet. SecondNet achieves scalability by distributing all the virtual-to-physical mapping, routing, and bandwidth reservation state in server hypervisors. Its port-switching based source routing (PSSR) further makes SecondNet applicable to arbitrary network topologies using commodity servers and switches. SecondNet introduces a centralized VDC allocation algorithm for bandwidth guaranteed virtual to physical mapping. Simulations demonstrate that our VDC allocation achieves high network utilization and low time complexity. Our implementation and experiments show that we can build SecondNet on top of various network topologies, and SecondNet provides bandwidth guarantee and elasticity, as designed.

Previously, the network must be congested by probing flow before the congestion-related parameters (such as background flow and available bandwidth) can be estimated, which lead to the inaccuracy and inefficiency of today's Internet. In this work, the authors study how to estimate network state without saturating the network. By introducing the queueing theory, this study proposes a novel packet time series analysis (PTSA) framework model, which can be used to estimate the congestion-related parameters without saturating the network. The accuracy and efficiency of PTSA are validated under NS-2 simulation environment. The performance of PTSA methodology is evaluated in Schooner test-bed with a special scenario. The analytical, simulative and experimental results show that PTSA framework is more efficient to estimate network state with less aggression and higher sensitivity than those methods that need to saturate the network.

In practice, because of their saturating affects, potentially all traditional transmission control protocol (TCP) congestion estimators effectively degrade the operational performance and efficiency of virtually all high-speed networks. Here the authors adopt a novel idea of using queueing-based estimator to estimate the TCP operating points ahead of their occurrence of network congestion. For this purpose the authors employ recent work of packet time-series analysis. In order to demonstrate the effectiveness of the idea, the authors embed the new estimator into several TCP variants in NS-2 and Linux kernel. For this evaluation the authors have programmed the simulation system to simulate a wide range of classic operating environments and setup many practical test-bed emulators for measurements. The authors results show that the idea of using a queueing-based estimator provides us far more earlier, sensitive and consistent estimation for the operating point than the classic loss/delay-based TCP traffic estimators. The authors then show that the new approach improves the overall estimation efficiency and provides higher performance and fairness when monitoring TCP traffic.

Shipping-container-based data centers have been introduced as building blocks for constructing mega-data centers. However, it is a challenge on how to interconnect those containers together with reasonable cost and cabling complexity, due to the fact that a mega-data center can have hundreds or even thousands of containers and the aggregate bandwidth among containers can easily reach tera-bit per second. As a new inner-container server-centric network architecture, BCube [9] interconnects thousands of servers inside a container and provides high bandwidth support for typical traffic patterns. It naturally serves as a building block for mega-data center.

This paper presents BCube, a new network architecture specifically designed for shipping-container based, modular data centers. At the core of the BCube architecture is its server-centric network structure, where servers with multiple network ports connect to multiple layers of COTS (commodity off-the-shelf) mini-switches. Servers act as not only end hosts, but also relay nodes for each other. BCube supports various bandwidth-intensive applications by speeding-up one-to-one, one-to-several, and one-to-all traffic patterns, and by providing high network capacity for all-to-all traffic.

Botnets are emerging threat with hundreds of millions of computers infected. A study shows that about 40% of all computers connected to the internet in the world are infected bots and controlled by attackers. This article is a survey of recent advances in botnet research. The survey classifies the botnet research into three areas: understanding botnets, detecting and tracking botnets, and defending against botnets. While botnets are widespread, the research and solutions for botnets are still in their infancy. The paper also summarizes the existing research and proposes future directions for botnet research.

A large number of overlay protocols have been developed, almost all of which assume each host has two-way communication capability. However, this does not hold as the deployment of firewalls and Network Address Translators (NAT) is widespread in the current Internet, which is a challenge to the design and implementation of overlay models and protocols. In this paper, we present Potential-based Multicast Tree Algorithm (PMTA) to enhance the multicast tree construction in presence of connectivity restricted hosts. We evaluate PMTA and previous multicast tree protocols based on real Internet end-to-end delay datasets. According to evaluation results, PMTA outperforms those protocols in terms of all metrics. PMTA reduces ARDP by 26%, and it also results in 23%-54% reduction in average overlay latencies. As the results suggest, PMTA can build efficient and effective multicast tree and is suitable for Internet multicast applications in the presence of connectivity restricted hosts.

Packet forwarding prioritization (PFP) in routers is one of the mechanisms commonly available to network administrators. PFP can have a significant impact on the performance of applications, the accuracy of measurement tools' results and the effectiveness of network troubleshooting procedures. Despite their potential impact, no information on PFP settings is readily available to end users. In this paper, we present an end-to-end approach for packet forwarding priority inference and its associated tool, POPI. This is the first attempt to infer router packet-forwarding priority through end-to-end measurement. Our POPI tool enables users to discover such network policies through the monitoring and rank classification of loss rates for different packet types. We validated our approach via statistical analysis, simulation, and wide-area experimentation in PlanetLab. As part of our wide-area experiments, we employed POPI to analyze 156 random paths across 162 PlanetLab nodes. We discovered 15 paths flagged with multiple priorities, 13 of which were further validated through hop-by-hop loss rates measurements. In addition, we surveyed all related network operators and received responses for about half of them confirming our inferences.

At the TCP sender side, the arrival of an ack packet always triggers the sender to send data packets, which establishes a correspondency between the arrived ack packet and the sent data packets. In a TCP connection, the correspondency between every ack packet and its corresponding data packets forms a sequence. This sequence characterizes the sender's behavior. In this paper, we propose a method to estimate this correspondency sequence from the dump trace measured at the receiver side. Because many possible correspondency sequences can be constructed based on the trace, the problem here is an estimation problem, which is to select a most possible one from those candidate sequences. The method proposed first eliminates some candidates that violate basic TCP congestion behavior. Then, it chooses the most possible one among the remaining sequences using the statistical characteristics of delays between the acks and their corresponding data packets under maximum-likelihood criterion. The method can work in the condition when the TCP connection experiences various network delay and loss, and it applies to TCP senders of different versions. Simulations and Internet experiments have been performed to validate the method.