arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2306.04743cs.DBcs.AIcs.CL
收藏

ScienceBenchmark:用于评估自然语言到SQL系统的复杂真实世界基准

ScienceBenchmark: A Complex Real-World Benchmark for Evaluating Natural Language to SQL Systems

Zurich University of Applied Sciences(苏黎世应用科学大学);Athena Research Center(雅典娜研究中心)

Yi Zhang, Jan Deriu, George Katsogiannis-Meimarakis, Catherine Kosten, Georgia Koutrika, Kurt Stockinger

AI总结:

本文提出ScienceBenchmark,一个基于三个真实世界领域特定科学数据库的复杂NL-to-SQL基准,结合专家标注与GPT-3合成数据,揭示现有顶尖系统在复杂真实场景下性能显著下降的挑战。

摘要:

自然语言到SQL系统(NL-to-SQL)最近在自然语言到SQL查询翻译的准确性方面取得了显著提升。这一改进源于基于Transformer的语言模型的出现,以及Spider基准——评估NL-to-SQL系统的事实标准——的普及。顶尖的NL-to-SQL系统准确率已达到85%。然而,Spider主要包含简单的数据库,表、列和条目数量较少,这并不能反映真实场景。此外,具有领域特定内容的复杂真实世界数据库几乎没有以NL/SQL对形式存在的训练数据,导致现有NL-to-SQL系统在这些数据库上表现不佳。在本文中,我们介绍了ScienceBenchmark,这是一个新的复杂NL-to-SQL基准,针对三个真实世界、高度领域特定的数据库。针对这一新基准,SQL专家和领域专家为每个领域创建了高质量的NL/SQL对。为了获取更多数据,我们使用GPT-3生成的合成数据扩展了少量人工生成的数据。我们表明,我们的基准极具挑战性,因为在Spider上表现最好的系统在我们的基准上性能非常低。因此,挑战是多方面的:为高度复杂的领域创建NL-to-SQL系统,同时使用少量手工制作的训练数据并用合成数据加以增强。据我们所知,ScienceBenchmark是第一个基于复杂真实世界科学数据库设计的NL-to-SQL基准,包含由领域专家仔细验证的具有挑战性的训练和测试数据。

英文摘要:

Natural Language to SQL systems (NL-to-SQL) have recently shown a significant increase in accuracy for natural language to SQL query translation. This improvement is due to the emergence of transformer-based language models, and the popularity of the Spider benchmark - the de-facto standard for evaluating NL-to-SQL systems. The top NL-to-SQL systems reach accuracies of up to 85\%. However, Spider mainly contains simple databases with few tables, columns, and entries, which does not reflect a realistic setting. Moreover, complex real-world databases with domain-specific content have little to no training data available in the form of NL/SQL-pairs leading to poor performance of existing NL-to-SQL systems. In this paper, we introduce ScienceBenchmark, a new complex NL-to-SQL benchmark for three real-world, highly domain-specific databases. For this new benchmark, SQL experts and domain experts created high-quality NL/SQL-pairs for each domain. To garner more data, we extended the small amount of human-generated data with synthetic data generated using GPT-3. We show that our benchmark is highly challenging, as the top performing systems on Spider achieve a very low performance on our benchmark. Thus, the challenge is many-fold: creating NL-to-SQL systems for highly complex domains with a small amount of hand-made training data augmented with synthetic data. To our knowledge, ScienceBenchmark is the first NL-to-SQL benchmark designed with complex real-world scientific databases, containing challenging training and test data carefully validated by domain experts.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑