We describe IDEA, an API designed specifically for the parallel processing of large spatial datasets on a cluster. Because such datasets present special challenges for efficient I/O and communication, it is especially valuable to provide an API that frees the user from the burden of partitioning the data among the processors. IDEA allows the user to address a communication to neighboring blocks of data, rather than processes or nodes. In addition to being very natural for the user, this data-centric view allows communication to a data block before it has been assigned a process. This is a key ability when handling data sets larger than the aggregate memory capacity of the cluster, since the dataset must be processed in a piecewise fashion.

High performance parallel computing infrastructures, such as computing clusters, have recently become freely available for scientific researchers to solve problems of unprecedented scale through data parallelization. However scientists are not necessarily skilled in writing efficient parallel code, especially when dealing with spatial datasets. Two important performance issues involved are the heavy I/O costs and the communication overhead. To address this issue, we are developing an scheme that helps scientists realize I/O friendly and scalable data parallelization for spatial computation.

Disk and network latency must be taken into account when applying parallel computing to large multidimensional datasets because they can hinder performance by reducing the rate at which data can be fed to the compute nodes. Existing methods aggregate some number of data requests from cluster nodes to improve overall performance by reducing the number of latency penalties. However, an even more significant reduction can be achieved by taking advantage of prior knowledge of the access pattern expressed as an iteration. Within the context of the granite scientific database system, we created a new iteration aware data distribution system that accelerates data transfer between a data server and the client cluster. This system reduces both disk and network latency by transforming a large number of small requests into a small number of large requests that fill an n-dimensional cache block on the cluster head node