arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2004.04487cs.CL
收藏

在语言模型中注入数值推理技巧

Injecting Numerical Reasoning Skills into Language Models

Tel Aviv University(特拉维夫大学);Allen Institute for AI(艾伦人工智能研究所)

Mor Geva, Ankit Gupta, Jonathan Berant

摘要:

已知大型预训练语言模型(LM)编码大量的语言信息。然而,高级推理技能,如数值推理,很难只从语言建模目标中学习。因此,现有的数值推理模型使用了灵活性有限的专门架构。在这项工作中,我们证明了数值推理可以自动生成数据,因此可以通过生成大量数据并在多任务设置中进行训练,将这种技能注入到预先训练的LM中。我们表明,在此数据上对我们的模型GenBERT进行预训练,可以显着提高DROP(49.3 $\rightarrow$ 72.3 F1)的性能,达到与具有可比大小的最先进模型相匹配的性能,同时使用简单且通用的编码器-解码器架构。此外,GenBERT可以很好地推广到数学应用题数据集,同时在标准RC任务上保持高性能。我们的方法提供了一个通用的配方注入技能到大型预先训练的LM,只要技能是服从自动数据增强。

英文摘要:

Large pre-trained language models (LMs) are known to encode substantial amounts of linguistic information. However, high-level reasoning skills, such as numerical reasoning, are difficult to learn from a language-modeling objective only. Consequently, existing models for numerical reasoning have used specialized architectures with limited flexibility. In this work, we show that numerical reasoning is amenable to automatic data generation, and thus one can inject this skill into pre-trained LMs, by generating large amounts of data, and training in a multi-task setup. We show that pre-training our model, GenBERT, on this data, dramatically improves performance on DROP (49.3 $\rightarrow$ 72.3 F1), reaching performance that matches state-of-the-art models of comparable size, while using a simple and general-purpose encoder-decoder architecture. Moreover, GenBERT generalizes well to math word problem datasets, while maintaining high performance on standard RC tasks. Our approach provides a general recipe for injecting skills into large pre-trained LMs, whenever the skill is amenable to automatic data augmentation.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑