Execution Accuracy and Exact Set Match are two predominant metrics for evaluating the functional correctness of SQL queries in modern Text-to-SQL tasks. However, both metrics have notable limitations: Exact Set Match fails when queries are functionally equivalent but syntactically different, while Execution Accuracy is prone to false positives due to inadequately prepared test databases, which can be costly to create, particularly in large-scale industrial applications. To overcome these challenges, we propose a novel graph-based metric, FuncEvalGMN, that effectively overcomes the deficiencies of the aforementioned metric designs. Our method utilizes a relational operator tree (ROT), referred to as RelNode, to extract rich semantic information from the logical execution plan of SQL queries, and embed it into a graph. We then train a graph neural network (GNN) to perform graph matching on pairs of SQL queries through graph contrastive learning. FuncEvalGMN offers two highly desired advantages: (i) it requires only the database schema to derive logical execution plans, eliminating the need for extensive test database preparation, and (ii) it demonstrates strong generalization capabilities on unseen datasets. These properties highlight FuncEvalGMN’s robustness as a reliable metric for assessing functional correctness across a wide range of Text-to-SQL applications.

DevOps (i.e., Development and Operations) is a growing concept, which aims to make building, testing, and releasing software faster, more frequent, and more reliable by automating the software delivery and architectural change process. Since DevOps is a relatively new concept, we seek to understand the hot topics in DevOps and the challenges encountered so far. We use data from Stack Overflow (SO), the largest developer question-and-answer platform, to understand the interests and difficulties of DevOps. First, we collected all DevOps-related posts from 2009 to 2022 based on SO tags. Then we used the Latent Dirichlet Allocation model to identify topics, and manually analyzed time trends and hot topics. The results indicate that: (1) DevOps-related issues can be divided into four categories: Container, Pipeline, Configuration, and Deployment; (2) The issues that get more attention are DevOps foundation issues, cloud deployment failure issues, Kubernetes technology usage issues, and DevOps complex environment issues, while the most difficult issues to solve are CI/CD-related issues; (3) DevOps has been introduced since 2009 and the number of questions on the SO platform is growing, especially between 2014 and 2020; (4) The percentage of DevOps-related questions with no accepted answers on the SO platform reached 56.3%. Our results show that problems are evident in terms of complex environments and the lack of experts. Also, developers need to pay more attention to basic theoretical knowledge when they are new to DevOps.