With the development of modern cloud platforms, an increasing number of users are migrating their data analysis tasks to the cloud. Cloud platforms offer a “pay-as-you-go” model, prompting users to focus on both performance and resource costs. Existing query optimization methods primarily address query performance while neglecting resource costs. Mapping queries to their resource consumption is a complex task. To tackle this challenge, we propose a novel learning-based query resource recommendation method called LORE. LORE efficiently and accurately estimates the optimal resources for queries by leveraging dual information from SQL query statements and query execution plans. We model SQL queries and execution plans as directed acyclic graphs and utilize graph neural networks to derive comprehensive representations. To capture the dependencies among all nodes involved in data transmission within an execution plan, we assign path weights to the dependency edges of each node. Our approach integrates data distribution information and captures both direct and indirect dependencies among plan nodes while avoiding unnecessary redundant computations. Experimental results demonstrate that, compared to traditional and other learning-based methods, the LORE model achieves higher accuracy in predicting the optimal resources for queries.

The efficiency of query processing is highly affected by execution plans and allocated resources in the Spark SQL big data processing engine. However, the cost models for Spark SQL are still based on hand-crafted rules. The learning-based cost models have been proposed for relational databases, but it does not consider the effect of the available resources. To address this, we propose a resource-aware deep learning model that can automatically predict the execution time of query plans based on historical data. To train our model, we embed the query execution plans based on the query plan tree and extract features from the allocated resources. A deep learning model with adaptive attention mechanisms is then trained to predict the execution time of query plans. The experiments show that our deep cost model can achieve higher accuracy in predicting the execution time of query plans compared to traditional rule-based methods and relational database learning-based optimizers.