The high performance storage system (HPSS) provides scalable hierarchical storage management (HSM), archive, and file system services. Its design, implementation and current dominant use are focused on HSM and archive services. It is also a general-purpose, global, shared, parallel file system, potentially useful in other application domains. When HPSS design and implementation began over a decade ago, scientific computing power and storage capabilities at a site, such as a DOE national laboratory, was measured in a few 10 s of gigaops, data archived in HSMs in a few 10 s of terabytes at most, data throughput rates to an HSM in a few megabytes/s, and daily throughput with the HSM in a few gigabytes/day. At that time, the DOE national laboratories and IBM HPSS design team recognized that we were headed for a data storage explosion driven by computing power rising to teraops/petaops requiring data stored in HSMs to rise to petabytes and beyond, data transfer rates with the HSM to rise to gigabytes/s and higher, and daily throughput with a HSM in 10 s of terabytes/day. This paper discusses HPSS architectural, implementation and deployment experiences that contributed to its success in meeting the above orders of magnitude scaling targets. We also discuss areas that need additional attention as we continue significant scaling into the future.

The National Storage Laboratory (NSL) was organized to develop, demonstrate and commercialize technology for the storage systems that will be the future repositories for our national information assets. Within the NSL four Department of Energy laboratories and IBM Federal Systems Company have pooled their resources to develop an entirely new High Performance Storage System (HPSS). The HPSS project concentrates on scalable parallel storage systems for highly parallel computers as well as traditional supercomputers and workstation clusters. Concentrating on meeting the high end of storage system and data management requirements, HPSS is designed using network-connected storage devices to transfer data at rates of 100 million bytes per second and beyond. The resulting products will be portable to many vendor’s platjorms. The three year project is targeted to be complete in 1995. This paper provides an overview of the requirements, design issues, and architecture of HPSS, as well as a description of the distributed, multi-organization industry and national laboratory HPSS project. 1: Storage system requirements and challenges The NationaI Storage Laboratory (NSL) has initiated a project to develop a next generation High Performance

It is argued that currently dominant, large-scale storage architectures, built around central, shared storage systems with CPU-connected devices, are reaching economic and technological limitations and no longer meet performance, capacity, and transparency requirements. Models of historical scientific and technological paradigm shifts are reviewed, and the reasons why the authors believe such a paradigm shift is underway in storage system architectures are discussed. The requirements to be met, important technical problems being investigated, such as network-connected devices, use of storage hierarchies, and system management, and the characteristics of the emerging large-scale, distributed, storage-architecture paradigm are described and illustrated by actual implementations and by standardization work under way in the IEEE Storage System Standards Working Group.>

An industry-led collaborative project, called the National Storage Laboratory (NSL), has been organized to investigate technology for storage systems that will be the future repositories for the national information assets. Lawrence Livermore National Laboratory through its National Energy Research Supercomputer Center (NERSC) is the operational site and the provider of applications. It is anticipated that the integrated testbed system will represent a significant advance in the technology for distributed storage systems capable of handling gigabyte class files at gigabit-per-second data rates. The NSL collaboration is undertaking research in four areas: network-attached storage; multiple, dynamic, distributed storage hierarchies; layered access to storage system services; and storage system management. An overview of the prototype storage system is given. Three application domains have been chosen to test and demonstrate the system's effect on scientific productivity; climatic models, magnetic fusion energy models, and digital imaging.>

With the advent of new high performance networks and distribution systems there is renewed interest in transport protocol designs that can support both request/response and stream styles of communication. The author examines Delta-t, a transport protocol designed to meet such goals. Delta-t's main contribution is in the area of connection management, where it achieves hazard-free connection management without explicit packet exchanges. The author reviews Delta-t's features and connection management in general and outlines some lessons useful for the implementation of high-performance networks.>

End-to-end transport protocols continue to be an active area of research and development involving (1) design and implementation of special-purpose protocols, and (2) reexamination of the design and implementation of general-purpose protocols. This work is motivated by the perceived low bandwidth and high delay, CPU, memory, and other costs of many current general-purpose transport protocol designs and implementations. This paper examines transport protocol mechanisms and implementation issues and argues that general-purpose transport protocols can be effective in a wide range of distributed applications because (1) many of the mechanisms used in the special-purpose protocols can also be used in general-purpose protocol designs and implementations, (2) special-purpose designs have hidden costs, and (3) very special operating system environments, overall system loads, application response times, and interaction patterns are required before general-purpose protocols are the main system performance bottlenecks.