arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2402.12554cs.CL
收藏

Archer:一个带算术、常识与假设推理的人工标注文本到SQL数据集

Archer: A Human-Labeled Text-to-SQL Dataset with Arithmetic, Commonsense and Hypothetical Reasoning

School of Informatics, University of Edinburgh(爱丁堡大学信息学院);Huawei Edinburgh Research Centre, CSI(华为爱丁堡研究中心)

Danna Zheng, Mirella Lapata, Jeff Z. Pan

AI总结:

本文提出双语文本到SQL数据集Archer,覆盖算术、常识与假设推理,包含中英文各1042个问题及521条SQL查询,实验显示顶尖模型执行准确率仅6.73%,为该领域研究带来显著挑战。

摘要:

我们提出了Archer,一个具有挑战性的双语文本到SQL数据集,专门针对复杂推理,包括算术推理、常识推理和假设推理。该数据集包含1,042个英文问题和1,042个中文问题,以及521条唯一SQL查询,覆盖20个领域中的20个英文数据库。值得注意的是,与现有公开数据集相比,该数据集展现出显著更高的复杂度。我们的评估表明,Archer对当前最先进模型的能力构成挑战,Spider排行榜上排名靠前的模型在Archer测试集上的执行准确率仅为6.73%。因此,Archer为该领域未来研究带来了重大挑战。

英文摘要:

We present Archer, a challenging bilingual text-to-SQL dataset specific to complex reasoning, including arithmetic, commonsense and hypothetical reasoning. It contains 1,042 English questions and 1,042 Chinese questions, along with 521 unique SQL queries, covering 20 English databases across 20 domains. Notably, this dataset demonstrates a significantly higher level of complexity compared to existing publicly available datasets. Our evaluation shows that Archer challenges the capabilities of current state-of-the-art models, with a high-ranked model on the Spider leaderboard achieving only 6.73% execution accuracy on Archer test set. Thus, Archer presents a significant challenge for future research in this field.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑