arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2402.12243cs.CL
收藏

理解Text-to-SQL中噪声的影响:对BIRD-Bench基准的考察

Understanding the Effects of Noise in Text-to-SQL: An Examination of the BIRD-Bench Benchmark

Linköping University(林雪平大学);Silo AI

Niklas Wretblad, Fredrik Gordh Riseby, Rahul Biswas, Amin Ahmadi, Oskar Holmström

AI总结:

本研究深入分析了BIRD-Bench基准中问题与金标准查询的噪声分布及其对模型可靠性的影响,发现在修正SQL后零样本基线超越最先进提示方法,强调可靠基准与噪声标签对开发Text-to-SQL方法的重要性。

摘要:

Text-to-SQL涉及将自然语言翻译为结构化查询语言(SQL),对于使非专家用户能够广泛访问结构化数据库至关重要。然而,由于众多因素,包括“噪声”的存在(如模糊问题和语法错误),为此类任务设计模型具有挑战性。本研究对广泛使用的BIRD-Bench基准中噪声的分布和类型,以及噪声对模型的影响进行了深入分析。虽然BIRD-Bench的创建是为了模拟脏数据和有噪声的数据库值,但其创建初衷并非包含问题和金标准查询中的噪声与错误。我们发现,问题和金标准查询中的噪声在数据集中普遍存在,在不同领域数量各异,且在噪声类型之间分布不均。不正确的金标准SQL查询的存在,进而生成不正确的金标准答案,对基准的可靠性产生了重大影响。令人惊讶的是,当在修正后的SQL查询上评估模型时,零样本基线超越了最先进的提示方法的性能。我们得出结论,信息丰富的噪声标签和可靠的基准对于开发能够处理不同类型噪声的新Text-to-SQL方法至关重要。所有数据集、标注和代码可在https://github.com/niklaswretblad/the-effects-of-noise-in-text-to-SQL获取。

英文摘要:

Text-to-SQL, which involves translating natural language into Structured Query Language (SQL), is crucial for enabling broad access to structured databases without expert knowledge. However, designing models for such tasks is challenging due to numerous factors, including the presence of 'noise,' such as ambiguous questions and syntactical errors. This study provides an in-depth analysis of the distribution and types of noise in the widely used BIRD-Bench benchmark and the impact of noise on models. While BIRD-Bench was created to model dirty and noisy database values, it was not created to contain noise and errors in the questions and gold queries. We found that noise in questions and gold queries are prevalent in the dataset, with varying amounts across domains, and with an uneven distribution between noise types. The presence of incorrect gold SQL queries, which then generate incorrect gold answers, has a significant impact on the benchmark's reliability. Surprisingly, when evaluating models on corrected SQL queries, zero-shot baselines surpassed the performance of state-of-the-art prompting methods. We conclude that informative noise labels and reliable benchmarks are crucial to developing new Text-to-SQL methods that can handle varying types of noise. All datasets, annotations, and code are available at https://github.com/niklaswretblad/the-effects-of-noise-in-text-to-SQL.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑