arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 1806.09029cs.CLcs.AIcs.DB
收藏

改进Text-to-SQL计算方法

Improving Text-to-SQL Evaluation Methodology

University of Michigan, Ann Arbor(密歇根大学安娜堡分校);Yale University(耶鲁大学)

Catherine Finegan-Dollak, Jonathan K. Kummerfeld, Li Zhang, Karthik Ramanathan, Sesh Sadasivam, Rui Zhang, Dragomir Radev

摘要:

为了提供信息,评估必须衡量系统如何推广到现实的不可见数据。我们确定的局限性,并提出改进,目前的文本到SQL系统的评估。首先,我们比较了人类生成的和自动生成的问题,描述了现实世界应用程序所需的查询属性。为了便于对多个数据集进行评估,我们发布了七个现有数据集和一个新的文本到SQL数据集的标准化和改进版本。其次,我们表明,目前的数据划分为训练集和测试集的措施的鲁棒性变化的方式提出的问题,但只有部分测试如何以及系统推广到新的查询,因此,我们提出了一个互补的数据集分割评估未来的工作。最后,我们展示了如何在评估过程中匿名变量的常见做法消除了任务的一个重要挑战。我们的观察突出了关键的困难,我们的方法可以有效地衡量未来的发展。

英文摘要:

To be informative, an evaluation must measure how well systems generalize to realistic unseen data. We identify limitations of and propose improvements to current evaluations of text-to-SQL systems. First, we compare human-generated and automatically generated questions, characterizing properties of queries necessary for real-world applications. To facilitate evaluation on multiple datasets, we release standardized and improved versions of seven existing datasets and one new text-to-SQL dataset. Second, we show that the current division of data into training and test sets measures robustness to variations in the way questions are asked, but only partially tests how well systems generalize to new queries; therefore, we propose a complementary dataset split for evaluation of future work. Finally, we demonstrate how the common practice of anonymizing variables during evaluation removes an important challenge of the task. Our observations highlight key difficulties, and our methodology enables effective measurement of future development.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑