Collaborative research in various scientific disciplines requires sup-port for scalable data management enabling the efficient correlation of globally distributed data sources. Motivated by the expected data rates of upcoming projects and a growing number of users, com-munities explore new data management techniques for achieving high throughput. Community-driven data grids deliver such high-throughput data distribution for scientific federations by partition-ing data according to application-specific data and query character-istics. Query hot spots are an important and challenging problem in this environment. Existing approaches to load-balancing from Peer-to-Peer (P2P) data management and sensor networks do not directly meet the requirements of a data-intensive e-science envi-ronment. In this paper, our contributions are partitioning schemes based on multi-dimensional index structures enabling communities to trade off data load balancing and handling query hot spots via splitting and replication. We evaluate the partitioning schemes with two typical kinds of data sets from the astrophysics domain and workloads extracted from Sloan Digital Sky Survey (SDSS) query traces and perform throughput measurements in real and simulated networks. The experiments demonstrate the improved workload distribution capabilities and give promising directions for the de-velopment of future community grids. 1.

E-science communities face huge data management challenges due to large existing data sets and expected data rates from forthcoming projects. Community-driven data grids provide a scalable, high-throughput oriented data management solution for scientific federations by employing domain-specific partitioning schemes and parallelism. In this paper, we present how community-driven data grids can adapt their query coordination strategies in the face of different typical submission scenarios. We explore the impact of submitting queries uniformly or having submission hot spots. By an extensive evaluation of five strategies on simulated and distributed setups, we show that some coordination strategies are preferable to others, regardless of submission skew. Based on our results, we can improve the usability and scalability of community-driven data grids for data-intensive applications.

Beyond already existing huge data volumes, e-science communities face major challenges in managing the anticipated data deluge of forthcoming projects. Community-driven data grids target at domain-specific federations and provide a distributed, collaborative data management by employing dominant data characteristics (e. g., data skew) and query patterns to optimize the overall throughput. By combining well-established techniques for data partitioning and replication with Peer-to-Peer (P2P) technologies we can address several challenging problems: data load balancing, handling of query hot spots, and the adaption to short-term burst as well as long-term load redistributions.

In federated Data Grids, individual institutions share their data sets within a community to enable collaborative data analysis. Data access needs to be provided in a scalable fashion since in most e-science communities, data sets do not only grow exponentially but also experience an increasing popularity. If data autonomy is retained, each individual institution has to ensure efficient access to its data. Analyzing application-specific data properties (such as data skew) or query characteristics (query patterns) and distributing data within Data Grids accordingly, allows for improved throughput for data-intensive applications and enables better load-balancing between shared resources. We propose a framework for investigating application-specific index structures for creating suitable partitioning schemes. We evaluate two variants of the well-known Quadtree data structure as well as the Zones approach, an index structure from the astrophysics domain, according to several criteria. Our framework improves data access within federated Data Grids and can be combined with well-established Grid methods as well as with more flexible P2P technologies.

The field of e-science currently faces many challenges. Among the most important ones are the analysis of huge volumes of scientific data and the connection of various sciences and communities, thus enabling scientists to share scientific interests, data, and research results. These issues can be addressed by processing large data volumes on-thefly in the form of data streams and by combining multiple data sources and making the results available in a network. In this paper, we demonstrate how e-science can benefit from research in computer science in the field of data stream management. In particular, we are concerned with processing multiple data streams in grid-based peer-to-peer (P2P) networks. We introduce spatial matching, which is a current issue in astrophysics, as a real-life e-science scenario to show how a data stream management system (DSMS) can help in efficiently performing associated tasks. We describe our new way of solving the spatial matching problem and present some evaluation results. In the course of the evaluation, our DSMS StarGlobe proves to be a valuable computing platform for astrophysical applications.