Many data centers extensively use virtual machines (VMs), which provide the flexibility to move workload among physical servers. VMs can be placed to maximize application performance, power efficiency, or even fault tolerance. However, VMs are typically repositioned without considering network topology, congestion, or traffic routes.

We demonstrate MPLS Traffic Engineering (MPLS-TE) and MPLS-based Virtual Private Networks (MPLS VPNs) using OpenFlow [1] and NOX [6]. The demonstration is the outcome of an engineering experiment to answer the following questions: How hard is it to implement a complex control plane on top of a network controller such as NOX? Does the global vantage point in NOX make the implementation easier than the traditional method of implementing it on every switch, embedded in the data plane? We implemented every major feature of MPLS-TE and MPLS-VPN in just 2,000 lines of code, compared to much larger lines of code in the more traditional approach, such as Quagga-MPLS. Because NOX maintains a consistent, up-to-date topology map, the MPLS control plane features are quite simple to implement. And its simplicity makes it easy to extend: We have easily added several new features; something a network operator could do to customize their network to meet their customers' needs.

Building on the success of unicast IP, IP Multicast adopted a simple, open, best-effort delivery model with many-to-many semantics. Despite several years of effort, a general, scalable and reliable end-to-end transport protocol analogous to TCP has proven elusive. Proposed solutions are either inflexible, or incur high control overhead. We present Lightweight Multicast Services (LMS), which enhance the IP Multicast model with simple forwarding services to facilitate scalable and efficient (compared to pure end-to-end) solutions to problems such as reliable multicast. In LMS, routers tag and steer control packets to preselected endpoints and perform fine-grain multicast to guide responses to a subset of the group without transport-level processing. LMS divides error control into transport and forwarding components, which allows the former to remain at the end-points while the latter is pushed to the routers, where it can be implemented very efficiently. The division is clean, resulting in significant gains in performance and scalability, while reducing application complexity. LMS reaches beyond reliable multicast to applications such as scalable collect, any-cast, and in general, any application that can benefit from a hierarchy congruent with the underlying topology.

Packet filters are rules for classifying packets based on their header fields. Packet classification is essential to routers supporting services such as quality of service (QoS), virtual private networks (VPNs), and firewalls. A filter conflict occurs when two or more filters overlap, creating an ambiguity in packet classification. Current techniques for resolving filter conflicts are based on prioritizing conflicting filters, and choosing the higher priority filter. We show that such ordering does not always work. Instead, we propose a new scheme for conflict resolution, which is based on the idea of adding resolve filters. Our main results are algorithms for detecting and resolving conflicts in a filter database. We have tried our algorithm on 3 existing firewall databases, and have found conflicts, which are potential security holes, in each of them.

Present-day Internet protocol routers typically employ monolithic operating systems that are not easily upgradable and extensible. With the rapid rate of protocol development it is becoming increasingly important to dynamically upgrade router software in an incremental fashion. We have designed and implemented a high-performance, modular, extended services router software architecture in the Net BSD operating system kernel. This architecture allows code modules, called plugins, to be dynamically added and configured at run time. One of the novel features of our design is the ability to bind different plugins to individual flows; this allows for distinct plugin implementations to seamlessly coexist in the same runtime environment. We achieve high performance through a carefully designed modular architecture, an innovative packet classification algorithm that is highly efficient, and by caching that exploits the flow-like characteristics of Internet traffic. Compared to a monolithic best effort kernel, our implementation requires an average increase in packet processing overhead of only 8%, or 600 cycles per packet when running on an Intel Pentium Pro at 233 MHz. By shortcutting the forward loop based on the per-flow state we establish, we can forward packets up to three times faster than the best effort kernel.

Link-striping algorithms are often used to overcome transmission bottlenecks in computer networks. Traditional striping algorithms suffer from two major disadvantages. They provide inadequate load sharing in the presence of variable-length packets, and may result in non-FIFO delivery of data. We describe a new family of link-striping algorithms that solves both problems. Our scheme applies to any layer that can provide multiple FIFO channels. We deal with variable-sized packets by showing how fair-queuing algorithms can be transformed into load-sharing algorithms. Our transformation results in practical load-sharing protocols, and shows a theoretical connection between two seemingly different problems. The same transformation can be applied to obtain load-sharing protocols for links with different capacities. We deal with the FIFO requirement for two separate cases. If a sequence number can be added to each packet, we show how to speed up packet processing by letting the receiver simulate the sender algorithm. If no header can be added, we show how to provide quasi FIFO delivery. Quasi FIFO is FIFO except during occasional periods of loss of synchronization. We argue that quasi FIFO is adequate for most applications. We also describe a simple technique for speedy restoration of synchronization in the event of loss. We develop an architectural framework for transparently embedding our protocol at the network level by striping IP packets across multiple physical interfaces. The resulting stripe protocol has been implemented within the NetBSD kernel. Our measurements and simulations show that the protocol offers scalable throughput even when striping is done over dissimilar links, and that the protocol synchronized quickly after packet loss. Measurements show performance improvements over conventional round-robin striping schemes and striping schemes that do not resequence packets. Some aspects of our solution have been implemented in Cisco's router operating system (IOS 11.3) in the context of Multilink PPP striping.

Providing quality of service (QoS) in an efficient and scalable manner in the Internet is a topic of active research. The technologies that have drawn the most attention are integrated services (IntServ), differential services (DiffServ) and label switching (MPLS). While these technologies are orthogonal in many respects and can coexist, they are all similar with respect to the fact that some or all the routers in such networks need to be programmed with state information associating data flows with QoS classes or priorities or labels. This paper describes a protocol called SSP (state setup protocol) which is designed to disseminate and manage this state information. In addition to a simple design which makes fast implementations feasible, SSP provides many features like state aggregation and third party signaling which makes SSP a suitable tool for network configuration, management and provisioning as well. A detailed discussion of SSP is presented along with numerous examples and performance measurements.

Retransmission based error control for large scale multicast applications is difficult because of implosion and exposure. Existing schemes (SRM, RMTP, TMTP LBRRM) have good solutions to implosion, but only approximate solutions to exposure. We present a scheme that achieves finer grain fault recovery by exploiting new forwarding services that allow us to create a dynamic hierarchy of receivers. We extend the IP multicast service model so that routers provide a more refined form of multicasting (which may be useful to other applications), that enables local recovery. The new services are simple to implement and do not require routers to examine or store application packets; hence, they do not violate layering. Besides providing better implosion control and less exposure than other schemes, our scheme integrates well with the current IP model, has small recovery latencies (it requires no back-off delays), and completely isolates group members from topology. Our scheme can be used with a variety of multicast routing protocols, including DVMRP and PIM. We have implemented our scheme in NetBSD Unix, using about 250 lines of new C-code. The implementation requires two new IP options, 4 additional bytes in each routing entry and a slight modification to IGMP reports. The forwarding overhead incurred by the new services is actually lower than forwarding normal multicast traffic.

No abstract available.

Present day routers typically employ monolithic operating systems which are not easily upgradable and extensible. With the rapid rate of protocol development it is becoming increasingly important to dynamically upgrade router software in an incremental fashion. We have designed and implemented a high performance, modular, extended integrated services router software architecture in the NetBSD operating system kernel. This architecture allows code modules, called plugins, to be dynamically added and configured at run time. One of the novel features of our design is the ability to bind different plugins to individual flows; this allows for distinct plugin implementations to seamlessly coexist in the same runtime environment. High performance is achieved through a carefully designed modular architecture; an innovative packet classification algorithm that is both powerful and highly efficient; and by caching that exploits the flow-like characteristics of Internet traffic. Compared to a monolithic best-effort kernel, our implementation requires an average increase in packet processing overhead of only 8%, or 500 cycles/2.1ms per packet when running on a P6/233.

Two important requirements for protocol implementations to be able to provide quality of service (QoS) guarantees within the endsystem are: (1) efficient processor scheduling for application and protocol processing and (2) efficient mechanisms for data movement. Scheduling is needed to guarantee that the application and protocol tasks involved in processing each stream execute in a timely manner and obtain their required share of the CPU. We have designed and implemented an operating system (OS) mechanism called the real-time upcall (RTU) to provide such guarantees to applications. The RTU mechanism provides a simple real-time concurrency model and has minimal overheads for concurrency control and context switching compared to thread-based approaches. To demonstrate its efficacy, we have built RTU-based transmission control protocol (TCP) and user datagram protocol (UDP) protocol implementations that combine high efficiency with guaranteed performance. For efficient data movement, we have implemented a number of techniques such as: (1) direct movement of data between the application and the network adapter; (2) batching of input-output (I/O) operations to reduce context switches; and (3) header-data splitting at the receiver to keep bulk data page aligned. Our RTU-based user-space TCP/Internet protocol (TCP/IP) implementation provides bandwidth guarantees for bulk data connections even with real-time and "best-effort" load competing for CPU on the endsystem. Maximum achievable throughput is higher than the NetBSD kernel implementation due to efficient data movement. Sporadic and small messages with low delay requirements are also supported using reactive RTUs that are scheduled with very low delay. We believe that ours is the first solution that combines good data path performance with application-level bandwidth and delay guarantees for standard protocols and OSs.

We are building a high performance 1.2 Gb/s ATM network interface chip called the APIC (ATM Port Interconnect Controller). In addition to borrowing useful ideas from a number of research and commercial prototypes, the APIC design embraces several innovative features, and integrates all of these pieces into a coherent whole. Some of the novel ideas incorporated in the APIC design include: protected DMA and protected I/O, which allow applications to queue data for transmission or reception directly from user-space, effectively bypassing the kernel. This argues for moving the entire protocol stack including the interface device driver into the user-space, thereby yielding better latency and throughput performance than kernel-resident implementations. Pool DMA when used with packet splitting, is a technique that can be used to build true zero-copy kernel-resident protocol stack implementations, using a page-remapping technique. Finally, orchestrated interrupts and interrupt demultiplexing are mechanisms used to reduce the frequency of interrupts issued by the APIC. Although many of these ideas have been developed in the context of an ATM network interface, we believe they are also applicable in other contexts. In particular, protected DMA and I/O are promising techniques for improving the performance of several different types of I/O devices.

This report describes an ATM multipoint connection caching strategy (AMCC) to control the explosive growth of traffic within the network and at an endpoint in a large distributed interactive simulation (DIS) application such as a battlefield simulation. For very large DIS applications with 100000 entities, the current method of broadcasting information among entities will no longer be feasible due to the computational and network bandwidth limitations. Our scheme divides the simulation space into grids and each grid square or a set of grid squares forms a multicast group. Entities join the groups within their perception range and thus, they receive state updates only from entities within their perception range. The AMCC utilizes the unique capabilities of our dynamic multipoint ATM signaling protocol, CMAP. We have implemented the AMCC and have successfully demonstrated a small DIS battlefield simulation application running over our ATM testbed.

Link striping algorithms are often used to overcome transmission bottlenecks in computer networks. Traditional striping algorithms suffer from two major disadvantages. They provide inadequateload sharing in the presence of variable length packets, and may result in non-FIFOdelivery of data. We describe a new family of link striping algorithms that solves both problems. Our scheme applies to any layer that can provide multiple FIFO channels. We deal with variable sized packets by showing how fair queuing algorithms can be transformed into load sharing algorithms. Our transformation results in practical load sharing protocols, and shows a theoretical connection between two seemingly different problems. The same transformation can be applied to obtain load sharing protocols for links with different capacities. We deal with the FIFO requirement for two separate cases. If a sequencenumber can be added to each packet, we show how to speed up packet processing by letting the receiver simulate the sender algorithm. If no header can be added, we show how to provide quasi-FIFO delivery. Quasi-FIFO is FIFO except during occasional periods of loss of synchronization. We argue that quasi-FIFOis adequatefor most applications. We also describe a simple technique for speedy restoration of synchronization in the event of loss. We develop an architectural framework for transparently embedding our protocol at the network level by striping IP packetsacross multiple physical interfaces. The resulting strIPe protocol has been implemented within the NetBSD kernel. Our measurementsand simulations showthat the protocol offers scalable throughputeven when striping is done over dissimilar links, and that the protocol synchronizes quickly after packet loss. Measurements show performance improvements over conventional round robin striping schemes and striping schemes that do not resequence packets. 1

This paper seeks to bridge the gap between theory and practice of real-time scheduling in the domain of high speed multimedia networking. We show that the strict preemptive nature of real-time scheduling leads to more context switching, and requires system calls for concurrency control. We present our scheduling scheme called rate-monotonic with delayed preemption (rmdp) and show how it reduces both these overheads. We then develop the analytical framework to analyze rmdp and other scheduling schemes that lie in the region between strict (immediate) preemption and no preemption. Our idealized scheduler simulation methodology accounts for the blocking introduced by these schemes under the usual assumption that the time for context switching and preemption is zero. We derive simpler schedulability tests for non-preemptive scheduling, and prove a variant of rate-monotonic scheduling that has fewer preemptions. Our measurements on Sparc and Pentium platforms, show that for the workloads we considered, Rmdp increases useful utilization by as much as 8%. Thus our scheduling policies have the potential to improve performance over existing methods.

Many new network applications demand interprocess communication (IPC) services that are not supported by existing transport protocol mechanisms. Large bandwidth-delay products of high-speed networks also render some of the existing error and flow control mechanisms less efficient. In particular, new error control schemes that can provide variable degrees of error recovery according to the application's requirements are desirable. This paper presents the design, evaluation, and implementation of an application-oriented error control-scheme that is aimed at supporting efficient IPC in high-speed networking environments. Our results indicate that the proposed error control scheme allows effective control of the trade-off between the amount of error an application can tolerate and the amount of delay it has to suffer.

We present the design of a high performance ATM host-network interface for multimedia workstations and servers. At Washington University, as part of an ARPA-sponsored gigabit local ATM testbed, we are building a prototype of this interface that can support a sustained aggregate bidirectional data rate of 2.4 Gbps. The centerpiece of our interface design is a custom chip called the APIC (ATM port interconnect controller). Multiple such chips can be interconnected to yield a desk-area network (DAN) which would serve as a high speed I/O interconnect for the host computer. This paper details the internal design of the APIC chip, and outlines some of its key features. Noteworthy among these are: connection caching, transmit pacing, cell batching, remote control, and support for AAL-0, AAL-5, multipoint, and loopback connections. We have chosen to defer to a later paper the details pertaining to several other features which provide support for zero-copy, improved interrupt handling, direct control of the chip from user-space, and efficient buffering and demultiplexing.

This paper describes research on new methods and architectures that enable the synergistic combination of IP and ATM technologies. We have designed a highly scalable gigabit IP router based on an ATM core and a set of tightly coupled general-purpose processors. This aItPm (pronounced "IP on ATM" or, if you prefer, "ip-attem") architecture provides flexibility in congestion control, routing, resource management, and packet scheduling.The aItPm architecture is designed to allow experimentation with, and fine tuning of, the protocols and algorithms that are expected to form the core of the next generation IP in the context of a gigabit environment. The underlying multi-CPU embedded system will ensure that there are enough CPU and memory cycles to perform all IP packet processing at gigabit rates. We believe that the aItPm architecture will not only lead to a scalable high-performance gigabit IP router technology, but will also demonstrate that IP and ATM technologies can be mutually supportive.

No abstract available.

This paper proposes using an explicit rate control together with a simple window control for flow control in high-speed networks with resource reservations. The design justifications, the evaluation study, and the implementation results are presented. In particular, the idea of hard ACK and soft ACK in the context of end-to-end flow control is explored through simulation and the results are discussed.>

Recently, a number of research groups have proposed connection-oriented access protocols that can provide a variable grade of service with performance guarantees on top of diverse networks. These connection-oriented internet protocols (COIPs) have performance tradeoffs. The authors have designed a COIP kernel (COIP-K), which can be used as a toolkit to implement any of the proposed COIPs, COIP-K features module interchange and incremental software support. The COIP-K implementation and its performance characteristics are presented.>

Results of a study that attempts to characterize the performance of the SunOS Inter-Process Communication (IPC) and TCP/IP protocols for distributed high-bandwidth applications are presented. In general, it is found that the performance of SunOS 4.0.3 IPC on Sparcstation 1 over the Ethernet is very good. An experiment has shown that average throughput rates of up to 7.5 Mb/s are achievable. Further investigation has shown that this rate is not limited by the TCP/IP performance, but by the driver performance. Therefore, a better driver implementation will increase throughput further. Some improvements to the default IPC are still possible. The performance of IPC for local processes is estimated to be about 43% of the memory bandwidth. The interaction of computation and communication is significant. An experiment has shown that the additional load on the machine greatly decreases throughput.>

Results of a study that characterizes the performance of SunOS Inter-Process Communication (IPC) and TCP/IP protocol implementation for distributed high-bandwidth applications are presented. Components studied include queuing in different layers, protocol control mechanisms (such as flow and error control), per-packet processing, buffer requirements, and interaction with the operating system. The Unix kernel and two public-domain tools for IPC measurement are reviewed.>

The primary goal of the Axon architecture is to support a high-performance data path delivering high network bandwidth directly to applications. The Axon network interface is described from the perspective of its simulation, and in particular and implementation of error and flow control in hardware. Background is provided on the simulation package that has been used to explore these mechanisms, and a brief overview is given of the Axon architecture is implemented by the simulator. The error control mechanism, the hardware design, and the functional and performance simulation results are outlined. The rate control scheme is described, along with its implementation and simulation.>

An introduction to the Axon architecture is presented. Axon is a host communication architecture for the distributed systems which can support interprocess communication with high throughput and low latency across a very high-speed internetwork. Network virtual storage (NVS) in the Axon host communication architecture for distributed applications is described. NVS extends segmented paged virtual storage management and address translation mechanisms to include segments located across an internetwork. This provides the ability to use efficiently the shared-memory paradigm in distributed systems for high-performance applications, such as scientific visualization and imaging.>

G.M. Parulkar (Comput. Commun. Rev., vol.20, no.1, p.18-43, Jan. 1990) previously proposed a very high-speed internet (VHSI) abstraction that provides a variable grade of service with performance guarantees on top of diverse networks. An improvement component of the VHSI abstraction is a novel multipoint congram-oriented high-performance internet protocol (MCHIP). Features of this protocol include the following: support for multipoint communication; the congram as the service primitive, incorporating strengths of both connection and datagram approaches; the ability to provide a variable grade of service with performance guarantees; and suitability for high-speed implementation. The authors introduce the VHSI abstraction, focusing on the description of MCHIP.>

The primary goal of the Axon architecture is to support a high-performance data path delivering VHSI (very high speed internetwork) bandwidth directly to applications. The significant features of Axon are: (1) an integrated design of host and network interface architecture, operating systems, and communication protocols; (2) a network virtual storage facility which includes support for virtual shared memory on loosely coupled systems; (3) a high-performance, lightweight object transport facility which can be used by both message-passing and shared-memory mechanisms; and (4) a pipelined network interface which can provide a path directly between the VHSI and host memory.>

A framework is formulated for high-speed communication in an environment comprising a mix of subnetworks with widely varying characteristics. It combines elements of recent work on high-speed packet switching and on interworking of computer networks. The work on high-speed packet switching is expected to lead to the development of large public networks capable of supporting applications ranging from low-speed data to voice, high-speed data and video. Interworking of existing computer networks with high-speed connection-oriented networks will allow them to be used effectively by applications that require the high performance levels such networks can provide.>

Noahnet is an experimental flood local area network with features such as high reliability and high performance. Noahnet uses a randomly connected graph topology with four to five interconnections per node and a flooding protocol to route messages.

Noahnet is a robust, highly available, high bandwidth local area network (LAN) architecture, in the implementation phase at the University of Delaware. Noahnet uses a distributed switch-oriented node topology, a flooding approach for message routing, and high bandwidth communication media. The purpose of this paper is to consider some of the important issues in detail such as: the various ways flood control can be implemented; their relative advantages/disadvantages; the functions of a node; and one possible implementation of a node in Noahnet. Expected performance of the Noahnet in comparison to Ethernet and token ring networks is also discussed. Though Noahnet uses more complex structure and protocol, we demonstrate that the overhead of using flooding in Noahnet is minimal and that a Noahnet node implementation is very simple and inexpensive. The paper concludes that Noahnet provides features such as robustness, high availability, high throughput, and high communication band width at almost no additional cost.