arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2601.04731cs.AIcs.CL
收藏

Miner:挖掘内在掌握以实现大推理模型的数据高效强化学习

Miner:Mining Intrinsic Mastery for Data-Efficient RL in Large Reasoning Models

Fudan University(复旦大学);Shanghai Jiao Tong University(上海交通大学);Shanghai Artificial Intelligence Laboratory(上海人工智能实验室)

Shuyang Jiang, Yuhao Wang, Ya Zhang, Yanfeng Wang, Yu Wang

AI总结:

本文提出Miner方法,通过利用策略内在不确定性作为自监督奖励信号,提升大推理模型的数据高效强化学习性能,实现状态-of-the-art结果。

摘要:

当前无批评者强化学习方法在训练正齐次提示(所有轨迹正确)时效率低下,导致零优势估计浪费轨迹。我们引入了一种简单而强大的解决方案来挖掘内在掌握(Miner),将策略的内在不确定性重新利用为自监督奖励信号,无需外部监督、辅助模型或额外推理成本。我们的方法开创了两个关键创新:(1)一种令牌级焦点信用分配机制,动态放大关键不确定令牌的梯度并抑制过于自信的令牌;(2)自适应优势校准,无缝整合内在和可验证奖励。在Qwen3-4B和Qwen3-8B基础模型上六个推理基准测试中,Miner在其他四种算法中取得最佳性能,相比GRPO在Pass@1上获得高达4.58的绝对增益,在Pass@K上获得6.66的增益。与其他针对探索增强的方法比较进一步揭示了这两个新提出的创新的优越性。这证明了潜在不确定性利用对于高效和可扩展推理模型强化学习是必要且充分的。代码可在https://github.com/pixas/Miner获取。

英文摘要:

Current critic-free RL methods for large reasoning models suffer from severe inefficiency when training on positive homogeneous prompts (where all rollouts are correct), resulting in waste of rollouts due to zero advantage estimates. We introduce a radically simple yet powerful solution to \uline{M}ine \uline{in}trinsic mast\uline{er}y (Miner), that repurposes the policy's intrinsic uncertainty as a self-supervised reward signal, with no external supervision, auxiliary models, or additional inference cost. Our method pioneers two key innovations: (1) a token-level focal credit assignment mechanism that dynamically amplifies gradients on critical uncertain tokens while suppressing overconfident ones, and (2) adaptive advantage calibration to seamlessly integrate intrinsic and verifiable rewards. Evaluated across six reasoning benchmarks on Qwen3-4B and Qwen3-8B base models, Miner achieves state-of-the-art performance among the other four algorithms, yielding up to \textbf{4.58} absolute gains in Pass@1 and \textbf{6.66} gains in Pass@K compared to GRPO. Comparison with other methods targeted at exploration enhancement further discloses the superiority of the two newly proposed innovations. This demonstrates that latent uncertainty exploitation is both necessary and sufficient for efficient and scalable RL training of reasoning models. Code is available at https://github.com/pixas/Miner.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑