arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2408.16293cs.CLcs.AIcs.LG
收藏

语言模型的物理学:第2.2部分,如何从小学数学问题的错误中学习

Physics of Language Models: Part 2.2, How to Learn From Mistakes on Grade-School Math Problems

CMU(卡内基梅隆大学);Meta FAIR;MBZUAI(穆罕默德·本·扎耶德人工智能大学)

Tian Ye, Zicheng Xu, Yuanzhi Li, Zeyuan Allen-Zhu

AI总结:

本文研究将“纠错”数据直接纳入预训练阶段以提升语言模型推理准确率,发现其优于使用等量无错误数据,并深入探讨了数据准备、掩码处理等多方面细节。

摘要:

语言模型在解决推理任务上展现出了卓越的性能;然而,即使是最强大的模型也偶尔会犯推理错误。最近,旨在提高推理准确率的活跃研究不断涌现,特别是通过多轮提示利用预训练语言模型来“自我纠正”其错误。在本文中,我们延续了这一研究方向,但重点在于理解将“纠错”数据直接纳入预训练阶段的有效性。该数据由错误的解答步骤及其紧随其后的纠正组成。使用合成数学数据集,我们展示了有前景的结果:与在同等数量的无错误数据上进行预训练相比,这种类型的预训练数据能帮助语言模型直接获得更高的推理准确率(即通过简单的自回归,无需多轮提示)。我们还深入探讨了许多细节,例如(1)这种方法与 beam search 有何不同,(2)如何准备此类数据,(3)是否需要对错误 token 进行掩码处理,(4)所需错误的数量,(5)此类数据是否可以推迟到微调阶段,以及许多其他问题。

英文摘要:

Language models have demonstrated remarkable performance in solving reasoning tasks; however, even the strongest models still occasionally make reasoning mistakes. Recently, there has been active research aimed at improving reasoning accuracy, particularly by using pretrained language models to "self-correct" their mistakes via multi-round prompting. In this paper, we follow this line of work but focus on understanding the usefulness of incorporating "error-correction" data directly into the pretraining stage. This data consists of erroneous solution steps immediately followed by their corrections. Using a synthetic math dataset, we show promising results: this type of pretrain data can help language models achieve higher reasoning accuracy directly (i.e., through simple auto-regression, without multi-round prompting) compared to pretraining on the same amount of error-free data. We also delve into many details, such as (1) how this approach differs from beam search, (2) how such data can be prepared, (3) whether masking is needed on the erroneous tokens, (4) the amount of error required, (5) whether such data can be deferred to the fine-tuning stage, and many others.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑