Online Scientific Applications (OSA) require statistical analysis of large multidimensional datasets. Towards this end, we have designed and developed a data storage and retrieval system, called ProDA, which deploys wavelet transform and provides fast approximate answers with progressively increasing accuracy in support of the OSA queries. ProDA employs a standard web-service infrastructure to enable remote users to interact with their data. These web-services enable wavelet transformation of large multidimensional datasets as well as inserting, updating, and exact, approximate and progressive querying of these datasets in the wavelet domain. We demonstrate the features of ProDA on a massive atmospheric dataset provided to us by NASA/JPL.

The Discrete Wavelet Transform is a proven tool for a wide range of database applications. However, despite broad acceptance, some of its properties have not been fully explored and thus not exploited, particularly for two common forms of multidimensional decomposition. We introduce two novel operations for wavelet transformed data, termed SHIFT and SPLIT, based on the properties of wavelet trees, which work directly in the wavelet domain. We demonstrate their significance and usefulness by analytically proving six important results in four common data maintenance scenarios, i.e., transformation of massive datasets, appending data, approximation of data streams and partial data reconstruction, leading to significant I/O cost reduction in all cases. Furthermore, we show how these operations can be further improved in combination with the optimal coefficient-to-disk-block allocation strategy. Our exhaustive set of empirical experiments with real-world datasets verifies our claims.