InfiniBand (IB) has become one of the most popular high-speed interconnects in High Performance Computing (HPC). The backpressure effect of credit-based link-layer flow control in IB introduces congestion spreading, which increases queueing delay and hurts application completion time. IB congestion control (IB CC) has been defined in IB specification to address the congestion spreading problem. Nowadays, HPC clusters are increasingly being used to run diverse workloads with a shared network infrastructure. The coexistence of messages transfers of different applications imposes great challenges to IB CC. In this paper, we re-exam IB CC through fine-grained experimental observations and reveal several fundamental problems. Inspired by our understanding and insights, we present a new receiver-driven congestion control for InfiniBand (RR CC). RR CC includes two key mechanisms: receiver-driven congestion identification and receiver-driven rate regulation, which empower eliminating both in-network congestion and endpoint congestion in one control loop. RR CC has much fewer parameters and requires no modifications to InfiniBand switches. Evaluations show that RR CC achieves better average/tail message latency and link utilization than IB CC under various scenarios.

With the rapid growth of hardware devices, a single host may have simultaneous connections that vary in network bandwidth and CPU processing capability as several orders of magnitude. State-of-art flow control mechanism, i.e., TCP auto-tuning, still needs to configure the maximum receive buffer, which cannot be applied to all connections in one host. In this paper, we reveal that improper receive buffer restrained by this configuration either (i) underutilizes the available network and CPU resources or (ii) occupies too much memory and then causes overall throughput collapse. To fully utilize resources with less memory occupancy, we present Receive Buffer Adaptive-regulating (RBA) algorithm, which regulates receive buffer according to the estimation of network bandwidth and receiver's processing capability. Testbed experiments show that RBA adapts to different scenarios and brings substantial performance improvement compared to TCP auto-tuning.

Large-scale data centers support various applications and process/issue terabits per second traffic from/to Internet. On the boundary of data center, the gateway needs to execute a series of network functions for each incoming packet. The Network Function Virtualization (NFV) technology leverages commodity servers to flexibly implement network functions. This solution provides satisfying processing and storage capability. However, state-of-the-art NFV platforms can merely process network functions at the line rate of 10~40Gbps. Supporting throughput of terabits per second requires dozens or even hundreds of servers operating exclusively for network functions, which is not only expensive but also difficult to maintain. On the other hand, programmable packet processing hardwares proposed in recent years offer a new platform for implementing network functions. They can execute user-defined packet processing logics at ultra-high line rate while containing limited processing and storage resources.

Many applications in distributed systems rely on underlying lossless networks to achieve required performance. Existing lossless network solutions propose different hop-by-hop flow controls to guarantee zero packet loss. However, another crucial problem called network deadlock occurs concomitantly. Once the system traps in a deadlock, a large part of network would be disabled. Existing deadlock avoidance solutions focus all their attentions on breaking the cyclic buffer dependency to eliminate circular wait (one necessary condition of deadlock). These solutions, however, impose many restrictions on network configurations and side-effects on performance.

Software Defined Network (SDN) enables flexible update of network functions with a well-defined abstraction between the control and the data plane. However, multiple active network functions with the same priority will potentially trigger conflicts among policies with overlapped flow space, causing the flow table explosion. In contrast to the local switch conflict resolution schemes proposed by previous works, this paper tackles the same problem from a different angle and resolves the policy conflict problem by coordinating all switches under a global centralized view. Specifically, we propose COnflict RAzor (CORA), which tremendously reduces the storage cost of conflicting policies leveraging the global network information obtained in the controller. The basic idea of CORA is migrating policies causing large explosions across the network if necessary, while keeping the semantics equivalence. We prove CORA's NP hardness and propose a heuristic to efficiently search a near-optimal policy migration strategy. Our experiments demonstrate that, CORA can effectively reduce the flow table storage occupation by at least 49% within less than 40 seconds.

Congestion control is an indispensable mechanism in the new trend of enhanced Ethernet as a unified fabric for traditional LAN, SAN, and high-performance computing networks. A congestion management framework for Converged Ethernet (CE) networks has been standardized by IEEE 802.1 Qau work group, and QCN is recommended as the congestion control scheme in the standard draft. QCN is heuristically designed for 1/10Gbps Ethernet without considering the impact of delays. Recent work find that QCN will encounter stability issues with feedback delays, and these issues will be more serious as Ethernet extends to 40/100Gbps and the delays become heterogeneous and time-varying. This work aims to mitigate the negative impact of delays on congestion control scheme in CE. Specially, considering the delays are heterogeneous and time-varying, we build a model for Converged Ethernet with the standard congestion management framework. The model provides a new congestion detector to estimate the real congestion status under the impact of delays and regards the heterogeneous and time-varying feature as disturbances. Leveraging the new congestion detector and tolerating the disturbance through the sliding mode control method, we design the Delay-tolerant Sliding Mode (DSM) congestion control scheme. Extensive simulations show that DSM outperforms other congestion control schemes when the Ethernet ranges from 1Gbps to 100Gbps and the delays are heterogeneous and time-varying.

Owing to Ethernet's low cost, high bandwidth and architecture openness, much attention has been paid to develop converged Ethernet to support both time-critical services and conventional communication services on a unified network infrastructure. The greatest challenge here is providing low and deterministic latency for time-critical packets. Recently, the IEEE time sensitive networking task group is launched to address it. However, their framework is complex and unsuitable for commodity switch architecture. In this paper, we propose a concise and efficient converged real-time Ethernet framework called XpressEth, which leverages Dual Preemption mechanism to minimize the delay of time-critical packets, and employs a lightweight Slot Assignment Scheduler to minimize the conflicts among time-critical packets at sources. XpressEth cuts off great burden from both forwarding and scheduling. The simulation results verify that XpressEth can provide ultra-low and deterministic latency for time-critical packets (1.024μ s per hop and zero jitter in 1Gbps network), which is 13× better than time sensitive networking solution, and the side-effect on conventional communication traffic is negligible.

High-precision time synchronization is critical for nowadays industrial Ethernet systems. Most existing time synchronization mechanisms are implemented based on packet communication. This interaction pattern, however, greatly limits their synchronizing frequency. In order to achieve microsecond-level synchronization precision, expensive high-quality oscillator is necessary for maintaining low clock skew under this long synchronization period (usually several seconds). Furthermore, packet processing introduces many nondeterministic variances (e.g. network stack overhead), which needs to be carefully eliminated. In this paper, we propose the brand-new Industrial Time Protocol (ITP). We deploy the entire ITP in the physical layer, so it eliminates most time uncertainties caused by network stack processing. Furthermore, ITP leverages the InterFrame Gap (IFG), which is the inherent interval between any two Ethernet frames, to carry the synchronization message. With this novel design, ITP can synchronize peer devices at very high frequency without degrading the goodput. The accuracy of ITP is bounded by 16ns for two adjacent devices with only intrinsic cheap oscillator. Furthermore, our theoretical analysis deduces that ITP guarantees 16N-nanosecond accuracy for N-hop network. We implement ITP design with NetFPGA. Experiments show that ITP can provide about 76-nanosecond accuracy for #hops=16 network under severe congestions. In addition, the design of ITP is scalable. It only consumes about 0.67% of logic cells in the low-end FPGA for supporting every ITP-aware port increase.