arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2603.20799cs.CLcs.LG
收藏

基于可验证奖励的强化学习训练不会提升大语言模型的通用问答能力:评估方法和一个简单解决方案

RLVR Training of LLMs Does Not Improve Thinking Ability for General QA: Evaluation Method and a Simple Solution

National Key Laboratory for Novel Software Technology(新型软件技术国家重点实验室);School of Artificial Intelligence, Nanjing University(南京大学人工智能学院);Huawei Technologies Co., Ltd.(华为技术有限公司)

Kaiyuan Li, Jing-Cheng Pang, Yang Yu

AI总结:

本文评估了基于可验证奖励的强化学习是否能提升大语言模型在通用问答任务中的推理能力,发现其效果不如可验证任务,并提出START方法提升问答质量和回答质量。

摘要:

基于可验证奖励的强化学习(RLVR)刺激大语言模型(LLMs)的思维过程,显著增强其在可验证任务上的推理能力。通常认为类似的收益应转移到通用问答(GQA),但这一假设尚未得到充分验证。为评估RLVR是否自动提升LLM在GQA上的性能,我们提出了一种跨代评估框架,通过将生成的思维上下文输入不同能力的LLMs来测量中间推理的质量。我们的评估得出令人失望的结论:在GQA任务中思维过程的有效性明显低于可验证任务,表明除了在可验证任务上训练外,还需对GQA进行显式训练。我们进一步发现,直接在GQA上进行RL训练效果不如RLVR。我们的假设是,尽管可验证任务需要稳健的逻辑链才能获得高奖励,但GQA任务往往允许通过捷径获得高奖励而无需培养高质量的思维。为了避免可能的捷径,我们引入了一种简单的方法,分离的思维和回答训练(START),首先只训练思维过程,使用定义在最终答案上的奖励。我们证明START在多个GQA基准和RL算法上同时提升了思维质量和最终答案质量。

英文摘要:

Reinforcement learning from verifiable rewards (RLVR) stimulates the thinking processes of large language models (LLMs), substantially enhancing their reasoning abilities on verifiable tasks. It is often assumed that similar gains should transfer to general question answering (GQA), but this assumption has not been thoroughly validated. To assess whether RLVR automatically improves LLM performance on GQA, we propose a Cross-Generation evaluation framework that measures the quality of intermediate reasoning by feeding the generated thinking context into LLMs of varying capabilities. Our evaluation leads to a discouraging finding: the efficacy of the thinking process on GQA tasks is markedly lower than on verifiable tasks, suggesting that explicit training on GQA remains necessary in addition to training on verifiable tasks. We further observe that direct RL training on GQA is less effective than RLVR. Our hypothesis is that, whereas verifiable tasks demand robust logical chains to obtain high rewards, GQA tasks often admit shortcuts to high rewards without cultivating high-quality thinking. To avoid possible shortcuts, we introduce a simple method, Separated Thinking And Response Training (START), which first trains only the thinking process, using rewards defined on the final answer. We show that START improves both the quality of thinking and the final answer across several GQA benchmarks and RL algorithms.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑