arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2109.04672cs.CLcs.AIcs.LG
收藏

文本到文本迁移模型的算术学习能力研究

Investigating Numeracy Learning Ability of a Text-to-Text Transfer Model

Arizona State University(亚利桑那州立大学)

Kuntal Kumar Pal, Chitta Baral

摘要:

基于transformer的预训练语言模型在大多数传统的NLP任务中取得了巨大的成功。但在需要数字理解的任务中,他们往往会遇到困难。一些可能的原因可能是标记器和预训练目标不是专门为学习和保持算术而设计的。在这里,我们研究了文本到文本的迁移学习模型(T5)学习算术的能力,该模型在传统的NLP任务中表现优于其前辈。我们考虑四个算术任务:计算,数量级预测,寻找最小值和最大值的一系列,排序。我们发现,虽然T5模型在插值设置中表现相当不错,但在所有四个任务的外推设置中,它们都相当挣扎。

英文摘要:

The transformer-based pre-trained language models have been tremendously successful in most of the conventional NLP tasks. But they often struggle in those tasks where numerical understanding is required. Some possible reasons can be the tokenizers and pre-training objectives which are not specifically designed to learn and preserve numeracy. Here we investigate the ability of text-to-text transfer learning model (T5), which has outperformed its predecessors in the conventional NLP tasks, to learn numeracy. We consider four numeracy tasks: numeration, magnitude order prediction, finding minimum and maximum in a series, and sorting. We find that, although T5 models perform reasonably well in the interpolation setting, they struggle considerably in the extrapolation setting across all four tasks.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑