arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2104.07307cs.CL
收藏

NT5?!训练T5进行数值推理

NT5?! Training T5 to Perform Numerical Reasoning

University of California(加利福尼亚大学);Google Research(谷歌研究院)

Peng-Jian Yang, Ying Ting Chen, Yuechan Chen, Daniel Cer

摘要:

文本数值推理(NRoT)提出了独特的挑战,现有的预训练目标没有很好地解决。我们探索了五个顺序训练计划,这些计划适应了NRoT的预训练T5模型。我们的最终模型改编自T5,但在三个数据集上进行了进一步的预训练,旨在加强NRoT和一般阅读理解所需的技能,然后在文本离散推理(DROP)数据集上进行微调。训练将DROP调整后的F1表现(以数字为中心的分数)从45.90提高到70.83。我们的模型接近GenBERT(72.4),这是一个自定义的BERT-Base模型,使用相同的数据集,具有更多的参数。我们表明,训练T5多任务框架与多个数值推理数据集的难度增加,良好的性能DROP可以实现,而无需手动工程分布式和符号模块之间的分区功能。

英文摘要:

Numerical reasoning over text (NRoT) presents unique challenges that are not well addressed by existing pre-training objectives. We explore five sequential training schedules that adapt a pre-trained T5 model for NRoT. Our final model is adapted from T5, but further pre-trained on three datasets designed to strengthen skills necessary for NRoT and general reading comprehension before being fine-tuned on the Discrete Reasoning over Text (DROP) dataset. The training improves DROP's adjusted F1 performance (a numeracy-focused score) from 45.90 to 70.83. Our model closes in on GenBERT (72.4), a custom BERT-Base model using the same datasets with significantly more parameters. We show that training the T5 multitasking framework with multiple numerical reasoning datasets of increasing difficulty, good performance on DROP can be achieved without manually engineering partitioned functionality between distributed and symbol modules.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑