Downloading an entire file is not practical for very large n-dimensional (n-d) datasets, especially if the region of interest (ROI) is small. It is therefore important to develop methods to allow researchers to remotely access n-d subsets of large datasets. Since researchers often wish to access a series of subsets, an awareness of the relationship between dataset storage organization and the application access pattern is important to guarantee effective performance. In our previous work, we reduced the impact of network latency costs even when efficient n-d access is available. This reduction in network latency costs exposes disk latency as an important performance factor. To address this new bottleneck and other problems, we propose the addition of n-d replicas to the data grid

Although processing speed, storage capacity and network bandwidth are steadily increasing, network latency remains a bottleneck for scientists accessing large remote data sets. This problem is most acute with n-dimensional data. Grid researchers have only recently begun to develop tools for efficient remote access to n-dimensional data sets. Within the context of the Granite Scientific Database system, we show that latency penalties can be dramatically reduced using explicit knowledge of a user's access pattern represented as an iterator. The iterator not only performs an n-dimensional iteration for the user, but also communicates the access pattern to Granite so that a prefetching cache can be constructed that is tuned to the user's access pattern. We experimentally evaluate a scenario for incorporating Granite's prefetching mechanism into the grid, demonstrating extraordinary performance gains. In light of these results, we describe planned additions to existing grid services to allow selection of datasets according to the user access pattern