arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2404.07017cs.CLcs.AI
收藏

通过自激励学习提升语言模型推理能力

Improving Language Model Reasoning with Self-motivated Learning

Research Center for Social Computing and Information Retrieval(社会计算与信息检索研究中心);Harbin Institute of Technology(哈尔滨工业大学);Huawei Noah's Ark Lab(华为诺亚方舟实验室)

Yunlong Feng, Yang Xu, Libo Qin, Yasheng Wang, Wanxiang Che

AI总结:

本文提出自激励学习框架,通过模型自动生成推理依据并利用正确性排序训练奖励模型进行强化学习,从而在无需高昂人工标注的情况下显著提升语言模型的推理能力。

摘要:

大规模高质量训练数据对于提升模型性能至关重要。在使用带有推理依据(推理步骤)的数据进行训练后,模型能够获得推理能力。然而,由于标注成本高昂,具有高质量推理依据的数据集相对稀缺。为了解决这一问题,我们提出了自激励学习框架。该框架激励模型自身在现有数据集上自动生成推理依据。基于多个推理依据之间正确性所固有的排序,模型学习生成更好的推理依据,从而获得更高的推理能力。具体而言,我们利用该排序训练一个奖励模型来评估推理依据的质量,并通过强化学习提升推理性能。Llama2 7B 在多个推理数据集上的实验结果表明,我们的方法显著提升了模型的推理能力,甚至在某些数据集上超越了 text-davinci-002。

英文摘要:

Large-scale high-quality training data is important for improving the performance of models. After trained with data that has rationales (reasoning steps), models gain reasoning capability. However, the dataset with high-quality rationales is relatively scarce due to the high annotation cost. To address this issue, we propose \textit{Self-motivated Learning} framework. The framework motivates the model itself to automatically generate rationales on existing datasets. Based on the inherent rank from correctness across multiple rationales, the model learns to generate better rationales, leading to higher reasoning capability. Specifically, we train a reward model with the rank to evaluate the quality of rationales, and improve the performance of reasoning through reinforcement learning. Experiment results of Llama2 7B on multiple reasoning datasets show that our method significantly improves the reasoning ability of models, even outperforming text-davinci-002 in some datasets.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑