Deployment scalability was introduced to capture a very general notion that is often a goal of shared infrastructure. It refers to the ability of the infrastructure's basic service to grow across many boundaries that might constrain it, through the acceptable application of resources, and while maintaining it at a standard level. In order to achieve deployment scalability the Internet architecture created a common infrastructure interface, or “spanning layer,” at the Network layer of the communication stack. This architecture was adopted to achieve two central goals: 1) The spanning layer virtualizes the variety of local services, enabling interoperability through the adoption of a common model; and 2) it provides an abstraction that hides the complex and dynamic topology and behavior of local infrastructure, thereby restricting the ability of clients to inspect or control local resources. The central design choice of the Internet Architecture is to make end-to-end datagram delivery the “bearer service” that defines its spanning layer. By contrast, Exposed Buffer Processing creates a more general platform by defining an underlay platform that provides the resources for implementing networking, storage, and computation. Exposed Buffer Processing implements a) a low-level “data plane” that provides fundamental persistence, transfer, and processing functionality and b) a higher-level programmable “control plane” that defines a variety of more global services.

To use heterogeneous and geographically distributed resources as a platform for parallel visualization is an intriguing topic of research. This is because of the immense potential impact of the work, and also because of its use of a full range of challenging technologies. In this work, we designed an execution environment for visualization of massive scientific datasets, using network functional units (NFU) for processing power, logistical networking for storage management and visualization cookbook library (vcblib) for visualization operations. This environment is based solely on computers distributed across the Internet that are owned and operated by independent institutions, while being openly shared for free. Those Internet computers are inherently of heterogeneous hardware configuration and running a variety of operating systems. Using 100 such processors, we have been able to obtain the same level of performance offered by a 64-node cluster of 2.2 GHz P4 processors, while processing a 75GBs subset of a cutting-edge simulation dataset. Due to its inherently shared nature, this execution environment for data-intensive visualization could provide a viable means of collaboration among geographically separated users.

We are interested in developing the infrastructural tools that allow a distributed data intensive computing environment to be shared by a group of collaborating but geographically separated researchers in an interactive manner, as opposed to a batch mode of operation. However, without advanced reservation, it is difficult to assure a certain level of performance on a large number of shared and heterogeneous servers. To achieve scalable parallel speedups in this scenario, we must closely integrate the management of computation and runtime data movement. In this paper, we first define the canonical scheduling problem for datasets distributed with k-way replication in the wide area. We then develop a dynamic co-scheduling algorithm that integrates the scheduling of computation and data movement. Using time-varying visualization as the driving application, we demonstrate that our co-scheduling approach improves not only application performance but also server utilization at a very reasonable cost.

To effectively share computation resources over the wide area network among a variety of data intensive applications, a scalable computation service needs to be provisioned. The end-to-end principles provide a scalable approach to the architecture of shared services on which these applications depend. We have shown the use of a best-effort network storage service, the Internet Backplane Protocol (IBP), for scalable data sharing by applying the end-to-end principles. This paper explores a way to scalable network computation by adhering to the end-to-end principles and discusses the costs to achieve scalability in our design

Scientific applications often need to access remotely located files, but many remote I/O systems lack standard APIs that allow efficient and direct access from application codes. This work presents MPI-IO/L, a remote I/O facility for MPI-IO using logistical networking. This combination not only provides high-performance and direct remote I/O using the standard parallel I/O interface but also offers convenient management and sharing of remote files. We show the performance trade-offs with various remote I/O approaches implemented in the system, which can help scientists identify preferable I/O options for their own applications. We also discuss how logistical networking could be improved to work better with parallel I/O systems such as ROMIO.

This paper describes a simple scheduling procedure for use in multicast data distribution within a logistical networking infrastructure. The goal of our scheduler is to generate a distribution schedule that will exploit the best network paths by using historic network performance information. A "spanning tree" is constructed between available logistical depots to help reduce the overall time of data movement. Our hypothesis is that we can generate appropriate schedules from historical network measurements. In order to evaluate the scheduling procedure we have employed the multicast operation used in the Internet backplane protocol (IBP) middleware suite. Investigation into the merits of such a scheduling procedure involved a control group that performs a broadcast to a set of logistical depots and an experimental group that is configured to perform a multicast via schedules generated based on historical network data. All testing was conducted on PlanetLab, a distributed network service testbed.

An exposed approach in computer service architecture is one that offers client software a primitive service whose semantics are closely based on the underlying physical infrastructure. The exposed approach relies on the client to build higher-level services, with more abstract semantics, out of such primitive tools using sophisticated compilation or run-time algorithms. Current approaches to reliable multicast focus on encapsulated algorithms for efficient retransmission of datagrams to sets of receivers that require them. These approaches include augmenting the primary multicast data channel with direct TCP connections or with secondary multicast channels for retransmissions, and on the possibility of retransmissions originating from nodes in the middle of the network. In this paper we offer an exposed approach to multicast that uses an underlying Logistical Networking infrastructure that makes possible the implementation of any of the current retransmission algorithms, as well as new strategies yet to be devised.

The need to provide remote visualization of large datasets with adequate levels of quality and interactively has become a major impediment to distributed collaboration in Computational Science.Although Image Based Rendering (IBR) techniques based on plenoptic functions have some important advantages over other approaches to this problem, they suffer from an inability to deal with issues of network latency and server load, due to the large size of the IBR databases they generate.Consequently, IBR techniques have been left largely unexplored for this purppose.In this paper we describe strategies for addressing these obstacles using Logistical Networking (LoN), which is a new and highly scalable approach to deploying storage as a shared communication resource.Leveraging LoN technology and infrastructure, we developed a remote visualization system based on concepts of light field rendering, an IBR method using a 4-D plenoptic function.Our system extends existing work on light fields by employing a modified method of parameterization and data organization that supports more efficient prefetching, caching and loss-less compression. Using this approach, we have been able to interactively browse multi-gigabyte, high-resolution light field databases across the wide area network at 30 frames per second.

In this work we present the Internet Backplane Protocol (IBP), a middleware created to allow the sharing of storage resources, implemented as part of the network fabric. IBP allows an application to control intermediate data staging operations explicitly. As IBP follows a very simple philosophy, very similar to the Internet Protocol, and the resulting semantic might be too weak for some applications, we introduce the exNode, a data structure that aggregates storage allocations on the Internet.

In this paper we present the Logistical Networking principles and the tools we developed to support our ideas. Logistical Networking is defined as the global scheduling and optimization of data movement, storage and computation based on a model that takes into account al the network's underlying physical resources. We introduce then the Internet Backplane Protocol (IBP), a middleware created to allow the sharing of storage resources, implemented as part of the network fabric, the exNode, a data structure that aggregates storage allocations on the Internet, and the L-Bone, a directory service of distributed storage resource.

This paper discusses the application of end-to-end design principles, which are characteristic of the architecture of the Internet, to network storage. While putting storage into the network fabric may seem to contradict end-to-end arguments, we try to show not only that there is no contradiction, but also that adherence to such an approach is the key to achieving true scalability of shared network storage. After discussing end-to-end arguments with respect to several properties of network storage, we describe the Internet Backplane Protocol and the exNode, which are tools that have been designed to create a network storage substrate that adheres to these principles. The name for this approach is Logistical Networking, and we believe its use is fundamental to the future of truly scalable communication.

Article Share on Enabling full service surrogates using the portable channel representation Authors: Micah Beck Innovative Computing Laboratory, Department of Computer Science, University of Tennessee, Knoxville, TN Innovative Computing Laboratory, Department of Computer Science, University of Tennessee, Knoxville, TNView Profile , Terry Moore Innovative Computing Laboratory, Department of Computer Science, University of Tennessee, Knoxville, TN Innovative Computing Laboratory, Department of Computer Science, University of Tennessee, Knoxville, TNView Profile , Leif Abrahamsson Lokomo Systems AB, Svärdvägen 27, aSE-182 33 Danderyd, Sweden Lokomo Systems AB, Svärdvägen 27, aSE-182 33 Danderyd, SwedenView Profile , Christophe Achouiantz Lokomo Systems AB, Svärdvägen 27, aSE-182 33 Danderyd, Sweden Lokomo Systems AB, Svärdvägen 27, aSE-182 33 Danderyd, SwedenView Profile , Patrick Johansson Lokomo Systems AB, Svärdvägen 27, aSE-182 33 Danderyd, Sweden Lokomo Systems AB, Svärdvägen 27, aSE-182 33 Danderyd, SwedenView Profile Authors Info & Claims WWW '01: Proceedings of the 10th international conference on World Wide WebMay 2001Pages 376–385https://doi.org/10.1145/371920.372091Published:01 April 2001Publication History 5citation379DownloadsMetricsTotal Citations5Total Downloads379Last 12 Months1Last 6 weeks0 Get Citation AlertsNew Citation Alert added!This alert has been successfully added and will be sent to:You will be notified whenever a record that you have chosen has been cited.To manage your alert preferences, click on the button below.Manage my AlertsNew Citation Alert!Please log in to your account Save to BinderSave to BinderCreate a New BinderNameCancelCreateExport CitationPublisher SiteGet Access

Checkpointing systems are a convenient way for users to make their programs fault-tolerant by intermittently saving program state to disk and restoring that state following a failure. The main concern with checkpointing is the overhead that it adds to running time of the program. This paper describes memory exclusion, an important class of optimizations that reduce the overhead of checkpointing. Some forms of memory exclusion are well-known in the checkpointing community. Others are relatively new. In this paper, we describe all of them within the same framework. We have implemented these optimization techniques in two checkpointers: libckpt, which works on Unix-based workstations, and CLIP, which works on the Intel Paragon. Both checkpointers are publicly available at no cost. We have checkpointed various long-running applications with both checkpointers and have explored the performance improvements that may be gained through memory exclusion. Results from these experiments are presented and show the improvements in time and space overhead. Copyright © 1999 John Wiley & Sons, Ltd.

The topic of intermediate languages for optimizing and parallelizing compilers has received muchattention lately. In this paper, we argue that any good representation of a program must havetwo crucial properties: first, it must be a data structure that can be rapidly traversed to determine dependence information, and second this representation must be a program in its own right, with a parallel, local model of execution. In this paper, we illustrate the importance of these points by examining algorithms for a standard optimization --- global constant propagation. We discuss the problems in working with current representations. Then, we propose a novel representation called the dependence flow graph which has each of the properties mentioned above. Weshow that this representation leads to a simple algorithm, based on abstract interpretation, for solving the constant propagation problem. Our algorithm is simpler than, and as efficient as, the best known algorithms for this problem. An interesting feature of our representation is that it naturally incorporates the best aspects of many other representations, including continuation-passing style, data and program dependence graphs, static single assignment form and dataflow program graphs.

Are imperative languages tied inseparably to the von Neumann model or can they be implemented in some natural way on data-flow architectures? In this paper, we show how imperative language programs can be translated into dataflow graphs and executed on a dataflow machine like Monsoon. This translation can exploit both fine-grain and coarse-grain parallelism in imperative language programs. More importantly, we establish a close connection between our work and current research in the imperative languages community on data dependences, control dependences, program dependence graphs, and static single assignment form. These results suggest that dataflow graphs can serve as an executable intermediate representation in parallelizing compilers.

Dynamic dataflow machines exploit parallelism among loop iterations by loop unraveling: all iterations of the loop are started together and operations in various iterations execute when their input data are present. Unbounded loop unraveling can strain the resources available on the machine and, in extreme cases, deadlock can occur due to overcommitment of resources. Previous efforts to address this problem have focused mainly on run-time mechanisms of debatable utility. Loop bounding, a compile-time technique, controls parallelism by permitting a fixed number of iterations to execute at one time. In this paper, we argue that loop bounding can lead to inefficient use of resources, and we propose an alternative way of compiling loops for overlapped execution of loop iterations. We introduce the notion of a stage decomposition of a loop, which defines a partition of the operations in a loop iteration into stages, and we show that the problem of choosing a stage decomposition for a particular loop can be tackled by applying static scheduling techniques like the ones used in generating code for VLIW machines. These techniques permit the compiler to allocate resources more skillfully than with loop bounding. The practical utility of stage decomposition remains to be tested on a real dataflow machine. In the absence of one, we describe how our schema could be implemented on the Monsoon dataflow machine being built at MIT.

The authors investigate the feasibility and efficiency of a parallel sort-merge algorithm by considering its implementation of the JASMIN prototype, a backend multiprocessor built around a fast packet bus. They describe the design and implementation of a parallel sort utility and present and analyze the results of measurements corresponding to a range of file sizes and processor configurations. The results show that using current, off-the-shelf technology coupled with a streamlined distributed operating system, three- and five-microprocessor configurations, provide a very cost-effective sort of large files. The three-processor configuration sorts a 100-Mb file in 1 hr which compares well to commercial sort packages available on high-performance mainframes. In additional experiments, the authors investigate a model to tune their sort software and scale their results to higher processor and network capabilities.>