Declustering is a well known technique to achieve high performance for queries on parallel databases. We propose novel General Disk Module (GDM) based declustering algorithms, GDM Cartesian and GDM Circle, for distributing uniformly distributed multidimensional datasets to parallel disks, for datasets of any dimension. We compare the performance of the new approaches with several existing declustering algorithms, using variable numbers of disks, and with variable shapes and dimensions of the datasets. Our results show that the new approaches significantly outperform the others for almost all configurations tested.

A training algorithm of hidden Markov model (HMM) using query-based learning is proposed and applied to the recognition of isolated digits in this paper. An efficient query learning procedure is designed to provide the good training data to the oracle in query-based learning at low cost. The proposed algorithm uses the concept that stems from the gradient based inversion algorithm of artificial neural networks. The proposed algorithm is compared with conventional training methods on isolated digit recognition problem. The results show that the proposed query-based HMM learning algorithm can decrease the recognition error rate up to 60% in our experiments.

The complexity of parallel I/O systems imposes significant challenge in managing and utilizing the available system resources to meet application performance, portability and usability goals. We believe that a parallel I/O system that automatically selects efficient I/O plans for user applications is a solution to this problem. We present such an automatic performance optimization approach for scientific applications performing collective I/O requests on multidimensional arrays. The approach is based on a high level description of the target workload and execution environment characteristics, and applies genetic algorithms to select high quality I/O plans. We have validated this approach in the Panda, parallel I/O library. Our performance evaluations on the IBM SP show that this approach can select high quality I/O plans under a variety of system conditions with a low overhead, and the genetic algorithm-selected I/O plans are in general better than the default plans used in Panda.

Fine grained data distributions are widely used to balance computational loads across compute processes in parallel scientific applications. When a fine grained data distribution is used in memory, performance of I/O intensive applications can be limited not only by disk speed but also by message passing, because a large number of small messages may be generated by the implementation strategy used in the underlying parallel file system or parallel I/O library. Combining (or packetizing) a set of small messages into a large message is generally known to speed up parallel I/O. However, overall I/O performance is affected not only by small messages but also by other factors like cyclic block size and interconnect characteristics. We describe small message combination and communication scheduling for fine grained data distributions in the Panda parallel I/O library and analyze I/O performance on parallel platforms having different interconnects: IBM SP2, IBM workstation cluster connected by FDDI and Pentium II cluster connected by Myrinet.

Parallel I/O systems typically consist of individual processors, communication networks, and a large number of disks. Managing and utilizing these resources to meet performance, portability and usability goals of applications has become a significant challenge. We believe that a parallel I/O system that automatically selects efficient I/O plans for user applications is a solution to this problem. In this paper, we present such an automatic performance optimization approach for scientific applications performing collective I/O requests on multidimensional arrays. Under our approach, an optimization engine in a parallel I/O system selects optimal I/O plans automatically without human intervention based on a description of the application I/O requests and the system configuration. To validate our hypothesis, we have built an optimizer that uses a rule-based and randomized search-based algorithms to select optimal parameter settings in Panda, a parallel I/O library for multidimensional arr...

Adaptive mesh refinement (AMR) is a promising computational approach that investigates a scientific problem using grids of different resolutions in different portions of the problem space. This allows the more "interesting" areas of the problem space to be investigated in more detail than other areas. We present efficient ways for AMR-type applications to do input/output on the IBM SP2. We propose a disk layout that minimizes I/O costs for checkpoint/restart operations and investigate several allocation approaches to decluster grid data to multiple disks to enhance I/O parallelism for timestep/visualization operations. The experiments on the SP2 show that our strategies can help to shorten the I/O time to meet the performance requirements of the two types of I/O operations.

Panda is a high performance library for accessing large multidimensional array data on secondary storage of parallel platforms and networks of workstations. When using Panda as the I/O component of a scientific application, H3expresso, on the IBM SP2 at Cornell Theory Center, we found that some nodes are more powerful with respect to I/O than others, requiring the introduction of load balancing techniques to maintain high performance. We expect that heterogeneity will also be a big issue for DBMSs or parallel I/O libraries designed for scientific applications running on networks of workstations, and the methods of allocating data to servers in these environments will need to be upgraded to take heterogeneity into account, while still allowing users to exert control over data layout. We propose such an approach to load balancing, under which we respect the user's choice of high level disk layout, but introduce automatic subchunking. The use of subchunks allows us to divide the very large chunks typically specified by the user's disk layout into more manageable size units that can be allocated to I/O nodes in a manner that fairly distributes the load. We also present two techniques for allocating subchunks to nodes, static and dynamic, and evaluate their performance on the SP2.

We present an analytical performance model for Panda, a library for synchronized i/o of large multidimensional arrays on parallel and sequential platforms, and show how the Panda developers use this model to evaluate Panda's parallel i/o performance and guide future Panda development. The model validation shows that system developers can simplify performance analysis, identify potential performance bottlenecks, and study the design trade-offs for Panda on massively parallel platforms more easily than by conducting empirical experiments. More importantly, we show that the outputs of the performance model can be used to help make optimal plans for handling application i/o requests, the first step toward our long-term goal of automatically optimizing i/o request handling in Panda.

Large multidimensional arrays are a common data type in high-performance scientific applications. Without special techniques for handling access to these arrays, I/O can easily become a large fraction of execution time for applications using these arrays, especially on parallel platforms. We show how to reduce the parallel I/O bottleneck for array data in closely-synchronized SPMD applications on distributed-memory platforms, through the use of server-directed I/O. This method allows array data requests on parallel platforms to be translated into long sequential disk reads and writes, while also minimizing the cost of rearranging data as they move between on-disk and in-memory schemas. We present experimental results from the implementation of server-directed I/O in Panda, showing that for I/O of large arrays, Panda utilizes nearly the maximum throughput of the underlying AIX file system on an IBM SP2. We also discuss Panda's user interface, an essential factor in Panda's high performance.