arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2109.05157cs.CL
收藏

探索跨域文本到SQL泛化的未充分探索的局限性

Exploring Underexplored Limitations of Cross-Domain Text-to-SQL Generalization

Queen Mary University of London(伦敦玛丽女王大学);UC Berkeley(加州大学伯克利分校);Jožef Stefan Institute(约瑟夫·斯蒂芬研究所)

Yujian Gan, Xinyun Chen, Matthew Purver

摘要:

最近,在研究用于在zero-shot跨域设置下将文本描述翻译成SQL查询的神经网络方面取得了重大进展。尽管在一些公共基准测试中取得了良好的性能,但我们观察到,现有的文本到SQL模型在面对不经常出现在训练数据中的领域知识时不会泛化,这可能会导致看不见的领域的预测性能较差。在这项工作中,我们调查的文本到SQL模型的鲁棒性时,问题需要很少观察到的领域知识。特别是,我们定义了五种类型的领域知识,并介绍了Spider-DK(DK是领域知识的缩写),这是一个基于Spider基准测试的人工数据集,用于文本到SQL的翻译。Spider-DK中的自然语言问题是从Spider中选取的,我们通过添加反映真实世界问题释义的领域知识来修改一些样本。我们证明,预测精度显着下降的样本,需要这样的领域知识,即使领域知识出现在训练集中,该模型提供了正确的预测相关的训练样本。

英文摘要:

Recently, there has been significant progress in studying neural networks for translating text descriptions into SQL queries under the zero-shot cross-domain setting. Despite achieving good performance on some public benchmarks, we observe that existing text-to-SQL models do not generalize when facing domain knowledge that does not frequently appear in the training data, which may render the worse prediction performance for unseen domains. In this work, we investigate the robustness of text-to-SQL models when the questions require rarely observed domain knowledge. In particular, we define five types of domain knowledge and introduce Spider-DK (DK is the abbreviation of domain knowledge), a human-curated dataset based on the Spider benchmark for text-to-SQL translation. NL questions in Spider-DK are selected from Spider, and we modify some samples by adding domain knowledge that reflects real-world question paraphrases. We demonstrate that the prediction accuracy dramatically drops on samples that require such domain knowledge, even if the domain knowledge appears in the training set, and the model provides the correct predictions for related training samples.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑