Many large applications require distributed computing for the sake of better performance and software systems that facilitate the development of such applications have attracted a great deal of attention. Modeling the application as distributed objects or components promises the benefits of better abstractions and increased software reuse. Using distributed object middleware (DOM) like CORBA (common object request broker architecture) looks promising, but most often one cannot afford its notorious inefficiency. We address the bandwidth bottleneck by extending highly efficient zero-copy communication architecture from the operating system through the middleware layers all the way to the application. In contrast to previous attempts on improving efficiency in CORBA we preserve the advantages of object oriented abstraction for the software design process and propose an efficient CORBA system that can handle bulk data transfers within the object request broker (ORB). Our prototype uses a clean separation of control-and data transfers within the ORB and for the ORB-to-ORB communication and manages to get rid of all inefficient buffering for certain types while still preserving the standard Internet interORB protocol (IIOP). It achieves the full performance that is only available with a strict zero-copy implementation across all layers between the operating system and the application.

Abstract Multicasting large amounts of data efficiently to all nodes of a PC clusteris an important operation. In the form of a partition cast it can be used to replicate entire software installations by cloning. Optimizing a partition cast for a given cluster of PCs reveals some interesting architectural tradeoffs, since the fastest solution does not only depend on the network speed and topology, but remains highly sensitive to other resources like the disk speed, the memory system performance and the processing power in the participating nodes. We present an analytical model that guides an implementation towards an optimal configuration for any given PC cluster. The model is validated by measurements on our cluster using Gigabit‐ and Fast‐Ethernet links. The resulting simple software tool, Dolly, can replicate an entire 2 GB Windows NT image onto 24 machines in less than 5 min. Copyright © 2002 John Wiley & Sons, Ltd.

As a novel approach to software maintenance in large clusters of PCs requiring multiple OS installations we implemented partition cloning and partition repositories as well as a set of OS independent tools for software maintenance using entire partitions, thus providing a clean abstraction of all operating system configuration states. We identify the evolution of software installations (different releases) and the customization of installed systems (different machines) as two orthogonal axes. Using this analysis we devise partition repositories as an efficient, incremental storage scheme to maintain all necessary partition images for versatile, large clusters of PCs. We evaluate our approach with a release history of sample images used in the Patagonia multi-purpose clusters at ETH Zurich including several Linux, Windows NT and Oberon images. The study includes quantitative data that shows the viability of the OS independent approach of working with entire partitions and investigates some relevant tradeoffs: e.g., between difference granularity and compression block size.

Cluster platforms offer good computational performance, but they still cannot utilize the potential of Gbit/s communication technology. While the speed of the Ethernet has grown to 1 Gbit/s, the functionality and the architectural support in the network interfaces has remained the same for more than a decade, so that the memory system becomes a limiting factor. To sustain the raw network speed in applications, a "zero-copy" network interface architecture would be required, but, for all widely used stacks, a last copy is required for the (de)fragmentation of the transferred network packets, since Ethernet packets are smaller than a page size. Correctly defragmenting packets of various communication protocols in hardware is an extremely complex task. We therefore consider a speculative defragmentation technique that can eliminate the last defragmenting copy operation in zero-copy TCP/IP stacks on existing hardware. The payload of fragmented packets is separated from the headers and stored in a memory page that can be mapped directly to its final destination in user memory. To evaluate our ideas, we integrated a network interface driver with speculative defragmentation into an existing protocol stack and added well-known page remapping and fast buffer strategies. Measurements indicate that we can improve the performance for a Gigabit Ethernet over a standard Linux 2.2 TCP/IP stack by a factor of 1.5-2 for uninterrupted burst transfers. Furthermore, our study demonstrates good speculation success rates for a database and a scientific application code on a cluster of PCs.