arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2606.28166cs.AI
收藏

具有可验证奖励的串联强化学习

Tandem Reinforcement Learning with Verifiable Rewards

University of Toronto(多伦多大学);EPFL(瑞士联邦理工学院洛桑分校)

Difan Jiao, Raghav Singhal, Robert West, Ashton Anderson

AI总结:

提出串联强化学习(TRL),通过强弱模型交替生成推理链并共同奖励,在保持独立推理能力的同时提升模型间兼容性和可读性。

摘要:

具有可验证奖励的强化学习(RLVR)显著提升了大语言模型的推理能力,在竞赛数学等领域达到专家甚至超人水平。然而,较弱智能体和人类能否实际利用这种能力远不确定,RLVR被记录为导致推理向不良模式漂移,如可读性差和语言混合。串联训练是最近引入的针对此兼容性问题的范式:训练更强的资深模型与冻结的较弱初级模型共同生成每次轨迹,两者作为团队获得奖励,从而推动资深模型以初级模型可跟随的方式推理。但该范式迄今仅在概念验证环境中展示,尚不清楚它是否能扩展到现代RLVR管道的长思维链。在这项工作中,我们提出串联强化学习(TRL),将串联训练范式引入RLVR。在TRL中,资深模型和冻结的初级模型随机交替共同生成推理,生成的推理获得奖励,并对资深模型应用标准GRPO损失。在竞赛数学上训练Qwen3-4B-Instruct,我们发现TRL在独立推理能力上与普通GRPO相当,同时从相同的轨迹结构中涌现出三个特性:与初级模型更强的交接鲁棒性、减少与初级模型的分布漂移,以及更易于初级模型理解的思维链。我们的结果展示了RLVR在多模型通信和人类兼容性方面具有实际收益的有前景的路径。

英文摘要:

Reinforcement learning with verifiable rewards (RLVR) has driven substantial gains in language-model reasoning. However, models fine-tuned with RLVR need not remain compatible with weaker partners: gains achieved alone may erode under collaboration, while policy drift may place reasoning beyond a weaker model's predictive reach. To address both problems, we introduce Tandem RLVR (TRLVR), bringing the recently proposed tandem-training paradigm beyond proof-of-concept settings and into modern RLVR. In TRLVR, a senior learns from rewarded trajectories co-generated with a frozen junior, while the standard GRPO objective is applied only to senior-emitted tokens. Training Qwen3-4B-Instruct on competition math, TRLVR performs on par with a matched GRPO senior when reasoning alone. We characterize compatibility through the complementary lenses of handoff robustness and junior legibility. Under reasoning-step handoffs, the TRLVR senior retains nearly all of its solo performance with the junior, reducing the communication tax to a negligible loss. The senior's reasoning is also more legible to the junior: its token distribution stays more closely anchored to the junior's, and the junior predicts the senior's chain-of-thought more readily token by token. An ablation that explicitly regularizes GRPO toward the junior does not recover the same combination of capability and compatibility. These results provide initial evidence that tandem rollout structure is a promising direction for keeping RLVR reasoning gains accessible to weaker partners.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑