This paper describes the lessons learned from the development and use of the eXtended Memory Management (XMM) subsystem in the Mach microkernel. XMM provides distributed memory management for single system image Unix on systems ranging from a few nodes to over 1000 nodes. XMM enables complete cross-node transparency at the Mach virtual memory interface in support of distributed file systems and distributed process execution, including making distributed shared memory available to applications. Our experience with XMM has revealed severe problems in scalability, performance, and complexity, leading us to conclude that significant portions of XMM's architecture and design should be viewed as failed experiments. Among the lessons we have learned from our experience are that copy on reference should be used instead of copy on write for memory management in distributed systems, and that virtual copy optimizations are inappropriate for distributed coherent virtual memory. © 1998 John Wiley & Sons, Ltd.

We have improved the performance of the Mach 3.0 operating system by redesigning its internal thread and interprocess communication facilities to use continuations as the basis for control transfer. Compared to previous versions of Mach 3.0, our new system consumes 85% less space per thread. Cross-address space remote procedure calls execute 14% faster. Exception handling runs over 60% faster.In addition to improving system performance, we have used continuations to generalize many control transfer optimizations that are common to operating systems, and have recast those optimizations in terms of a single implementation methodology. This paper describes our experiences with using continuations in the Mach operating system.