arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2401.02115cs.CL
收藏

使用 LLM 从候选中选择正确的 SQL 查询

Using LLM to select the right SQL Query from candidates

School of Computer Science, Peking University(北京大学计算机学院)

Zhenwen Li, Tao Xie

AI总结:

提出一种自动测试用例生成方法,通过生成数据库并用 LLM 预测预期执行结果,进而根据候选 SQL 查询的测试通过数和生成概率进行重排序,在 Spider 验证集上使最先进模型性能提升 3.6%。

摘要:

Text-to-SQL 模型可以生成候选 SQL 查询列表,最佳查询通常在候选列表中,但不在列表顶部。有效的重排序方法可以从候选列表中选择正确的 SQL 查询,提升模型性能。以往代码生成研究自动生成测试用例并用于重排序候选代码,但 Text-to-SQL 的自动测试用例生成仍是研究不足的领域。我们提出一种自动测试用例生成方法,首先生成数据库,然后使用 LLM 预测真实值,即真实 SQL 查询在该数据库上的预期执行结果。为降低 LLM 预测难度,我们通过实验探索生成易于 LLM 处理的数据库的方法,并设计易于理解的提示。基于该测试用例生成方法,我们提出一种重排序方法,从候选列表中选择正确的 SQL 查询。给定候选列表,我们的方法能生成测试用例,并根据候选查询在这些测试用例上的通过数量及其生成概率对候选列表进行重排序。在 Spider 验证集上的实验结果表明,应用我们的重排序方法后,一些最先进模型的性能可获得 3.6% 的提升。

英文摘要:

Text-to-SQL models can generate a list of candidate SQL queries, and the best query is often in the candidate list, but not at the top of the list. An effective re-rank method can select the right SQL query from the candidate list and improve the model's performance. Previous studies on code generation automatically generate test cases and use them to re-rank candidate codes. However, automatic test case generation for text-to-SQL is an understudied field. We propose an automatic test case generation method that first generates a database and then uses LLMs to predict the ground truth, which is the expected execution results of the ground truth SQL query on this database. To reduce the difficulty for LLMs to predict, we conduct experiments to search for ways to generate easy databases for LLMs and design easy-to-understand prompts. Based on our test case generation method, we propose a re-rank method to select the right SQL query from the candidate list. Given a candidate list, our method can generate test cases and re-rank the candidate list according to their pass numbers on these test cases and their generation probabilities. The experiment results on the validation dataset of Spider show that the performance of some state-of-the-art models can get a 3.6\% improvement after applying our re-rank method.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑