arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2602.03452cs.LGcs.AI
收藏

超越方差:通过稀有事件放大和双向配对实现高效的提示式强化学习可验证奖励

Beyond Variance: Prompt-Efficient RLVR via Rare-Event Amplification and Bidirectional Pairing

Beijing University of Post(北京邮电大学);Sichuan Agricultural University(四川农业大学);RapidAI Research(RapidAI研究院)

Xin Sheng, Jiaxin Li, Yujuan Pang, Ran Peng, Yong Ma

AI总结:

通过稀有事件放大和双向配对方法提升提示式强化学习可验证奖励的样本效率和转移性能。

摘要:

可验证奖励的强化学习(RLVR)在训练大语言模型进行确定性结果推理任务时是有效的。先前的工作表明RLVR在少量提示下工作,但提示选择通常仅基于训练准确性方差,导致优化方向不稳定和转移能力较弱。我们从机制层面重新审视提示选择,并认为一个有效的mini-batch应提供(i)一个可靠的正向锚点和(ii)来自罕见失败的明确负向学习信号。基于这一原则,我们提出了正负配对:在每次更新时,我们采样一个困难但可解的q⁺和一个容易但脆弱的提示q⁻(高成功率但不完美),其特征是在多个回放下具有低和高的经验成功率。我们进一步引入加权GRPO,该方法在配对层面重新加权二元结果,并利用组归一化优势来放大q⁺上的罕见成功,将其转化为尖锐的正向指导,同时将q⁻上的罕见失败转化为强负向惩罚。这种双向信号为成功和失败提供了信息性的学习反馈,提高了样本效率而不抑制探索。在Qwen2.5-Math-7B上,每次更新一个配对的mini-batch一致优于通过常用方差基于选择启发式方法选择两个提示的GRPO基线:AIME~2025 Pass@8从16.8提高到22.2,AMC23 Pass@64从94.0提高到97.0,同时与从1209个训练提示池中训练的大规模RLVR保持竞争力。在Qwen2.5-Math-7B-Instruct上也观察到类似收益。

英文摘要:

Reinforcement learning with verifiable rewards (RLVR) is effective for training large language models on deterministic outcome reasoning tasks. Prior work shows RLVR works with few prompts, but prompt selection is often based only on training-accuracy variance, leading to unstable optimization directions and weaker transfer. We revisit prompt selection from a mechanism-level view and argue that an effective minibatch should provide both (i) a reliable positive anchor and (ii) explicit negative learning signals from rare failures. Based on this principle, we propose \emph{positive--negative pairing}: at each update, we sample a hard-but-solvable $q^{+}$ and an easy-but-brittle prompt $q^{-}$(high success rate but not perfect), characterized by low and high empirical success rates under multiple rollouts. We further introduce Weighted GRPO, which reweights binary outcomes at the pair level and uses group-normalized advantages to amplify rare successes on $q^{+}$ into sharp positive guidance while turning rare failures on $q^{-}$ into strong negative penalties. This bidirectional signal provides informative learning feedback for both successes and failures, improving sample efficiency without suppressing exploration. On Qwen2.5-Math-7B, a single paired minibatch per update consistently outperforms a GRPO baseline that selects two prompts via commonly used variance-based selection heuristics: AIME~2025 Pass@8 improves from 16.8 to 22.2, and AMC23 Pass@64 from 94.0 to 97.0, while remaining competitive with large-scale RLVR trained from a pool of 1209 training prompts. Similar gains are observed on Qwen2.5-Math-7B-Instruct.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑