Pre-Calc:学习使用计算器提升语言模型的数值能力
Pre-Calc: Learning to Use the Calculator Improves Numeracy in Language Models
AI总结:
本文提出Pre-Calc预微调方法,通过让编码器或编码器-解码器语言模型学习使用计算器,在MAWPS等数据集上预训练,提升下游数值理解任务性能。
摘要:
语言中的定量和数值理解是教育和金融等许多领域的重要任务,但对语言模型来说仍然是一个具有挑战性的任务。虽然工具和计算器的使用已被证明有助于提高大型预训练仅解码器语言模型的数学推理能力,但这对于带有编码器的较小语言模型仍未得到探索。在本文中,我们提出了Pre-Calc,一种简单的预微调目标,即学习使用计算器,适用于仅编码器和编码器-解码器架构,分别表述为判别性和生成性任务。我们在MAWPS、SVAMP和AsDiv-A数据集上对BERT和RoBERTa进行判别性计算器使用的预训练,对Flan-T5进行生成性计算器使用的预训练,这提高了需要数值理解的下游任务的性能。我们的代码和数据可在https://github.com/calc-cmu/pre-calc获取。
英文摘要:
Quantitative and numerical comprehension in language is an important task in many fields like education and finance, but still remains a challenging task for language models. While tool and calculator usage has shown to be helpful to improve mathematical reasoning in large pretrained decoder-only language models, this remains unexplored for smaller language models with encoders. In this paper, we propose Pre-Calc, a simple pre-finetuning objective of learning to use the calculator for both encoder-only and encoder-decoder architectures, formulated as a discriminative and generative task respectively. We pre-train BERT and RoBERTa for discriminative calculator use and Flan-T5 for generative calculator use on the MAWPS, SVAMP, and AsDiv-A datasets, which improves performance on downstream tasks that require numerical understanding. Our code and data are available at https://github.com/calc-cmu/pre-calc.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。