arXiv 2407.145302024-07-09cs.DBcs.AI

FuncEvalGMN：基于图匹配网络的SQL功能正确性评估

FuncEvalGMN: Evaluating Functional Correctness of SQL via Graph Matching Network

ByteDance Inc.（字节跳动公司）

Yi Zhan, Yang Sun, Han Weng, Longjie Cui, Guifeng Wang, Jiajun Xie, Yu Tian, Xiaoming Yin, Boyi Liu, Dongchi Huang

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

提出基于图匹配网络的SQL功能正确性评估方法，利用关系运算符树和GNN预测，并贡献数据集Pair-Aug-Spider以克服传统指标局限。

摘要：

在本文中，我们提出了一种新颖的基于图的方法来评估SQL生成的功能正确性。用于评估SQL代码生成的常规指标，如基于匹配和基于执行的方法（例如，精确集合匹配和执行准确率），存在两个主要局限性。首先，前者无法有效评估功能正确性，因为不同的SQL查询可能具有相同的功能。其次，后者在评估中容易产生假阳性样本。我们提出的评估方法FuncEvalGMN不依赖于测试数据的充分准备，并且能够精确测试代码的功能正确性。首先，我们使用称为Relnode的关系运算符树（ROT）解析SQL，该树从逻辑执行的角度包含丰富的语义信息。然后，我们引入了一种基于GNN的方法来预测生成的SQL的功能正确性。该方法结合了全局位置嵌入，以解决传统图匹配框架中拓扑信息丢失的局限性。作为辅助贡献，我们提出了一种基于规则的匹配算法，即Relnode部分匹配（RelPM），作为基线。最后，我们贡献了一个数据集Pair-Aug-Spider，包含一个训练集和两个测试集，每个集包含成对的SQL代码，以模拟各种SQL代码评估场景。训练集和一个测试数据集侧重于使用大型语言模型（LLMs）进行代码生成，而另一个则强调SQL等价重写。

英文摘要：

In this paper, we propose a novel graph-based methodology to evaluate the functional correctness of SQL generation. Conventional metrics for assessing SQL code generation, such as matching-based and execution-based methods (e.g., exact set match and execution accuracy), are subject to two primary limitations. Firstly, the former fails to effectively assess functional correctness, as different SQL queries may possess identical functionalities. Secondly, the latter is susceptible to producing false positive samples in evaluations. Our proposed evaluation method, \texttt{FuncEvalGMN}, does not depend on the sufficient preparation of the test data, and it enables precise testing of the functional correctness of the code. Firstly, we parse SQL using a relational operator tree (ROT) called \textit{Relnode}, which contains rich semantic information from the perspective of logical execution.Then, we introduce a GNN-based approach for predicting the functional correctness of generated SQL. This approach incorporates global positional embeddings to address the limitations with the loss of topological information in conventional graph matching frameworks. As an auxiliary contribution, we propose a rule-based matching algorithm, Relnode Partial Matching (\texttt{RelPM}) as a baseline. Finally, we contribute a dataset, \texttt{Pair-Aug-Spider} with a training set and two testing sets, each comprising pairs of SQL codes to simulate various SQL code evaluation scenarios. The training set and one testing dataset focus on code generation using large language models (LLMs), while the other emphasizes SQL equivalence rewriting.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。

AI总结：

摘要：

英文摘要：

相关论文