arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2511.09158cs.AI
收藏

通过奖励模型实现高效推理

Efficient Reasoning via Reward Model

City University of Hong Kong(香港城市大学);Huawei Research(华为研究)

Yuhao Wang, Xiaopeng Li, Cheng Gong, Ziru Liu, Suiyun Zhang, Rui Liu, Xiangyu Zhao

AI总结:

针对大型推理模型(LRMs)过度思考导致的冗长回复问题,提出简洁性奖励模型(CRM)及含结果奖励与简洁性得分显式依赖的CRF,在数学基准上提升Qwen2.5-7B准确率8.1%并减少token长度19.9%,且泛化性良好。

摘要:

基于可验证奖励的强化学习(RLVR)已被证明能增强大型语言模型(LLMs)的推理能力,推动大型推理模型(LRMs)的发展。然而,DeepSeek-R1和OpenAI o1等LRMs常生成冗长回复,包含冗余或无关推理步骤——这一现象被称为过度思考——大幅增加计算成本。先前缓解该问题的尝试通常在奖励函数中加入长度惩罚,但我们发现它们常面临两个关键问题:长度坍缩和训练坍缩,导致次优性能。为解决这些问题,我们提出训练简洁性奖励模型(CRM)的流程,该模型对推理路径的简洁性打分。此外,我们引入名为简洁性奖励函数(CRF)的新型奖励公式,在结果奖励与简洁性得分间建立显式依赖,从而促进更有效且更高效的推理。从理论角度,我们从方差减少和收敛性改进的视角证明新奖励的优越性。在实践方面,在五个数学基准数据集上的大量实验证明该方法的有效性和token效率,在Qwen2.5-7B上实现8.1%的准确率提升和19.9%的回复token长度减少。此外,该方法对Llama和Mistral等其他LLMs具有良好泛化性。实现代码和数据集已公开供复现:https://anonymous.4open.science/r/CRM。

英文摘要:

Reinforcement learning with verifiable rewards (RLVR) has been shown to enhance the reasoning capabilities of large language models (LLMs), enabling the development of large reasoning models (LRMs). However, LRMs such as DeepSeek-R1 and OpenAI o1 often generate verbose responses containing redundant or irrelevant reasoning step-a phenomenon known as overthinking-which substantially increases computational costs. Prior efforts to mitigate this issue commonly incorporate length penalties into the reward function, but we find they frequently suffer from two critical issues: length collapse and training collapse, resulting in sub-optimal performance. To address them, we propose a pipeline for training a Conciseness Reward Model (CRM) that scores the conciseness of reasoning path. Additionally, we introduce a novel reward formulation named Conciseness Reward Function (CRF) with explicit dependency between the outcome reward and conciseness score, thereby fostering both more effective and more efficient reasoning. From a theoretical standpoint, we demonstrate the superiority of the new reward from the perspective of variance reduction and improved convergence properties. Besides, on the practical side, extensive experiments on five mathematical benchmark datasets demonstrate the method's effectiveness and token efficiency, which achieves an 8.1% accuracy improvement and a 19.9% reduction in response token length on Qwen2.5-7B. Furthermore, the method generalizes well to other LLMs including Llama and Mistral. The implementation code and datasets are publicly available for reproduction: https://anonymous.4open.science/r/CRM.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑