Emergence of new sensor technologies and increased parallelism has increased the size of scientific datasets dramatically in recent years. At the same time, research communities are demanding support for reproducible research, in which datasets and results are made available to other researchers. Such datasets do not change frequently, but future experiments can be derived from them.The volume of these datasets requires storage systems that scale across many machines, and also support highly selective queries to minimize the costs associated with unwanted data access.Decentralized Content Addressable Storage (CAS) systems such as the Inter-Planetary File System (IPFS) are promising as a way of disseminating datasets resulting from scientific research since they are scalable, the stored data is immutable, and versioning is straightforward. DS2, our prototype system for decentralized storage of spatial scientific data, bridges the gap between a scientist’s spatial view of a dataset and IPFS, the underlying decentralized filesystem. DS2 is intended to allow selective retrieval of data from very large datasets spanning many hosts.In this paper we investigate the feasibility of DS2 running on top of IPFS to store and retrieve large spatial scientific datasets. We also describe the decentralized metadata structures developed to support storage and retrieval of spatial datasets.

Because of the importance of Delaunay Triangulation in science and engineering, researchers have devoted extensive attention to parallelizing this fundamental algorithm. However, generating unstructured meshes for extremely large point sets remains a barrier for scientists working with large scale or high resolution datasets.

Scientific datasets are steadily growing in size, due to increasing resolution and scale. Unstructured meshes are essential to certain fields of engineering and science, but they present special challenges for efficient access and processing. The work described in this paper accelerates range queries for very large unstructured meshes using the GPU. Prior work in the area introduced a preprocessing phase that partitions large unstructured meshes in order to improve locality in storage and memory. Here, we apply the computational power and bandwidth of GPUs to the partitioning problem, significantly reducing preprocessing time. In order to keep the GPU busy, we have to overcome the poor locality of the original unstructured mesh. Toward this end, we developed our own approach to unstructured mesh I/O, called Direct Load. We show that Direct Load significantly outperforms a typical LRU cache. Our ultimate goal is to accelerate range queries. Our preprocessing steps allow us to parallelize range query processing with relatively simple GPU code. Experimental results show that our implementation outperforms the serial implementations by 4x for preprocessing and over 100 χ for range queries.

Scientific datasets today are often far too large to fit into a single machine's memory or even a single disk. Partitioning multidimensional arrays across several machines or disks has become increasingly necessary. However, relatively little work has been done for unstructured grids composed of a collection of simplicial cells. Our previous work investigated partitioning unstructured grids at the disk level and its effect on overall system performance. In this paper, we build upon prior work by investigating the effect of an in-core partitioning performed on top of the existing disk level partitioning. The granularity of in-core partitioning has varying effect on the overall system performance. Based on our test results, we propose a formula for choosing an effective partitioning for large unstructured grids to facilitate fast data retrieval. We also examine the performance benefits of declustering unstructured grids across several disks. Given this declustered dataset, we describe and explore a parallel data retrieval method that takes advantage of prior knowledge of a user access pattern. Our test results demonstrate very significant performance gains.

Due to the increasing quality of instruments and availability of computational resources, the size of spatial scientific datasets has been steadily increasing. However, much of the research on efficient storage and access to spatial datasets has focused on large multidimensional arrays. In contrast, unstructured datasets consisting of collections of simplices (e.g. triangles or tetrahedra) present special challenges that have received less attention. Data values found at the vertices of the simplices may be dispersed throughout a datafile, producing especially poor disk locality. In this paper, we address this important problem of poor locality in two major ways. First, we reorganize the unstructured dataset to improve locality in both the dataset space and in the data file on disk using a specialized chunking approach that maintains the spatial neighborhood relationships inherent in the unstructured data. This reorganization produces significant gains in performance by reducing the number of accesses made to the data file. Second, we extend our previous work and describe a prefetching method that takes advantage of prior knowledge of the user's access pattern. Applying this prefetching method to unstructured data produces further performance gains over and above the gains seen from reorganization alone.