arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2510.09535cs.CLcs.AI
收藏

通过推理塑形缓解过度思考

Mitigating Overthinking through Reasoning Shaping

State Key Laboratory of Multimedia Information Processing(多媒体信息处理国家重点实验室);School of Computer Science, Peking University(北京大学计算机学院)

Feifan Song, Shaohang Wei, Bofei Gao, Yejie Wang, Wen Luo, Wei Li, Linli Yao, Weimin Xiong, Liang Chen, Tianyu Liu, Houfeng Wang

AI总结:

本文提出步骤级方法GRSP,通过跨片段簇的长度感知加权机制规范推理,在保持准确性的同时显著提升token效率并稳定RL训练。

摘要:

受验证器奖励强化学习(RLVR)增强的大型推理模型(LRMs)在问题求解中展现出强大能力,但常引发过度思考:冗长发散的推理导致计算成本膨胀。RLVR中先前的惩罚设计虽能减少token消耗,却往往损害模型性能,这源于token级监督的过度简单。本文认为监督粒度在平衡效率与准确性方面起关键作用,并提出组相对片段惩罚(GRSP),一种用于规范推理的步骤级方法。初步分析表明推理片段与token消耗及模型性能高度相关,因此我们设计了一种跨片段簇的长度感知加权机制。大量实验表明,GRSP在不严重损害准确性的情况下实现了卓越的token效率,尤其是在难题上优势明显。此外,GRSP稳定了RL训练,并在不同模型规模间有效扩展。

英文摘要:

Large reasoning models (LRMs) boosted by Reinforcement Learning from Verifier Reward (RLVR) have shown great power in problem solving, yet they often cause overthinking: excessive, meandering reasoning that inflates computational cost. Prior designs of penalization in RLVR manage to reduce token consumption while often harming model performance, which arises from the oversimplicity of token-level supervision. In this paper, we argue that the granularity of supervision plays a crucial role in balancing efficiency and accuracy, and propose Group Relative Segment Penalization (GRSP), a step-level method to regularize reasoning. Since preliminary analyses show that reasoning segments are strongly correlated with token consumption and model performance, we design a length-aware weighting mechanism across segment clusters. Extensive experiments demonstrate that GRSP achieves superior token efficiency without heavily compromising accuracy, especially the advantages with harder problems. Moreover, GRSP stabilizes RL training and scales effectively across model sizes.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑