TAPEX:通过学习神经SQL执行器进行表格预训练
TAPEX: Table Pre-training via Learning a Neural SQL Executor
摘要:
语言模型预训练的最新进展通过利用大规模非结构化文本数据取得了巨大的成功。然而,由于缺乏大规模高质量的表格数据,在结构化表格数据上应用预训练仍然是一个挑战。在本文中,我们提出了TAPEX,以表明表预训练可以通过在合成语料库上学习神经SQL执行器来实现,该语料库是通过自动合成可执行的SQL查询及其执行输出来获得的。TAPEX通过引导语言模型在多样化、大规模、高质量的合成语料库上模仿SQL执行器来解决数据稀缺的挑战。我们在四个基准数据集上评估了TAPEX。实验结果表明,TAPEX的性能大大优于以前的表预训练方法,并在所有这些方法上都取得了新的最先进的结果。这包括弱监督的WikiSQL表示准确率提高到89.5%(+2.3%),WikiTableQuestions表示准确率提高到57.5%(+4.8%),SQA表示准确率提高到74.5%(+3.5%),TabFact准确率提高到84.2%(+3.2%)。据我们所知,这是第一项通过合成可执行程序利用表预训练并在各种下游任务上实现新的最先进结果的工作。我们的代码可以在https://github.com/microsoft/Table-Pretraining上找到。
英文摘要:
Recent progress in language model pre-training has achieved a great success via leveraging large-scale unstructured textual data. However, it is still a challenge to apply pre-training on structured tabular data due to the absence of large-scale high-quality tabular data. In this paper, we propose TAPEX to show that table pre-training can be achieved by learning a neural SQL executor over a synthetic corpus, which is obtained by automatically synthesizing executable SQL queries and their execution outputs. TAPEX addresses the data scarcity challenge via guiding the language model to mimic a SQL executor on the diverse, large-scale and high-quality synthetic corpus. We evaluate TAPEX on four benchmark datasets. Experimental results demonstrate that TAPEX outperforms previous table pre-training approaches by a large margin and achieves new state-of-the-art results on all of them. This includes the improvements on the weakly-supervised WikiSQL denotation accuracy to 89.5% (+2.3%), the WikiTableQuestions denotation accuracy to 57.5% (+4.8%), the SQA denotation accuracy to 74.5% (+3.5%), and the TabFact accuracy to 84.2% (+3.2%). To our knowledge, this is the first work to exploit table pre-training via synthetic executable programs and to achieve new state-of-the-art results on various downstream tasks. Our code can be found at https://github.com/microsoft/Table-Pretraining.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。