arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2610.24663cs.CLcs.LG
收藏

强化学习为推理带来了什么:基础模型不会做出的几个决策

What RL Brings to Reasoning: A Few Decisions the Base Model Would Not Make

University of California, San Diego(加利福尼亚大学圣迭戈分校);Meituan(美团);The Chinese University of Hong Kong, Shenzhen(香港中文大学(深圳))

Siyu Zhang, Yuquan Lu, Yoshua Y. Li, Yifan Ding, Kun Zeng, Yu Huo, Yifu Guo, Rongxiang Weng, Xunliang Cai, Xiaoying Tang

AI总结:

该研究以Qwen3-4B-Base为对象,通过生成时干预发现,借用RL模型与基础模型贪心选择不同的词元,可恢复大部分RL训练带来的数学推理增益,后续决策的贡献大于起始词元,且稀疏替换能兼顾增益与多样性。

摘要:

带可验证奖励的强化学习(RLVR)大幅提升了数学推理能力,但它为基础模型增添了什么仍不清楚。我们通过对Qwen3-4B-Base进行一系列生成时干预来研究这一问题,在120道AIME问题上测量每次干预能恢复其经RL训练的对应模型增益的比例。采用相同8192词元输出限制的幂采样,除贪心解码外无增益。然而,仅在两个模型的贪心选择不同的位置(约占生成位置的4%)借用RL模型的词元,可恢复90.5%的pass@1增益,在MATH-500的三个模型对上也有类似恢复效果。我们发现,即使是单个起始词元也有显著影响:强制基础模型以“Alright”开头可恢复35%的RL增益。不过仅靠起始词元不够,在相同起始词元后进行后续替换,可使pass@1再提升5.2个百分点,而我们测试的插入“Wait”等反思词的固定规则无法重现这一效果。稀疏替换在保持接近RL模型pass@1的同时,保留了基础模型的多样性。在k=256时,它解决了67.5%的问题,显著多于RL模型(61.4%)和pass@1匹配的完整响应混合模型(61.45%)。该比较基于相同尝试次数而非相同计算量,因为rl-replace在每个解码步骤都会查询两个模型。这些结果表明,起始词元可恢复部分RL增益,但更大贡献来自后续决策,可在保留基础模型多样性的同时借用这些决策。代码可在该https URL获取。

英文摘要:

Reinforcement learning with verifiable rewards (RLVR) substantially improves mathematical reasoning, but what it adds to the base model remains unclear. We investigate this through a sequence of generation-time interventions on Qwen3-4B-Base, measuring how much of its RL-trained counterpart's gain each intervention recovers on 120 AIME problems. Power sampling with the same 8,192-token output limit brings no gain beyond greedy decoding. Yet borrowing the RL model's token only where the two models' greedy choices differ, roughly 4% of generated positions, recovers 90.5% of the pass@1 gain, with similar recovery across three model pairs on MATH-500. We find that even a single opening token has a substantial effect: forcing the base model to begin with "Alright" recovers 35% of the RL gain. The opening alone is insufficient, however. Later replacements after the same opening add another 5.2 percentage points to pass@1, and the fixed rules we test for inserting reflective words such as "Wait" do not reproduce this benefit. Sparse replacement also retains much of the base model's diversity while staying close to the RL model at pass@1. At k=256, it solves 67.5% of the problems, significantly more than the RL model (61.4%) and a mixture of whole responses matched in pass@1 (61.45%). The comparison is at equal attempt counts rather than equal compute, since rl-replace consults both models at every decoding step. These results show that the opening can recover a meaningful part of the RL gain, but a larger contribution comes from later choices, which can be borrowed while retaining much of the base model's diversity. Code is available at https://github.com/CyrusZhang04/WhatRLBrings.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

相关深度报道

↑