arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2607.07976cs.CLcs.AIcs.LG
收藏

当不合理的令牌得到强化时:大语言模型强化学习的尾部感知信用校准

Right Answer, Wrong Tokens: Tail-Aware Credit Calibration for RLVR

Johns Hopkins University(约翰霍普金斯大学);Rice University(里士满大学);Workato(Workato公司);University of North Carolina at Charlotte(北卡罗来纳州夏洛特大学)

Xiuyi Lou, Zicheng Xu, Yu-Neng Chuang, Hoang Anh Duy Le, Zhaozhuo Xu, Guanchu Wang, Vladimir Braverman

AI总结:

研究大语言模型强化学习中统一信用分配的问题,提出尾部感知信用校准方法TACO,通过计算尾部风险分数校准信用,抑制不良正更新,实验证明该方法优于基线,提升训练稳定性。

摘要:

强化学习(RL)在提升大语言模型(LLMs)推理能力方面取得显著成功。然而,广泛使用的无评论家RL方法依赖统一信用分配,对所有令牌一视同仁。我们识别出这种设计的关键失败模式——正信用污染:上下文错误的低概率尾部令牌在同一轨迹中与合理令牌获得相同正信用,导致错误推理行为被不加区分地强化。为缓解此问题,我们提出尾部感知信用校准(TACO)方法,通过计算尾部风险分数来校准统一信用分配,抑制不良正更新。实验结果表明TACO持续优于GRPO风格基线,提高了训练稳定性。

英文摘要:

Although reinforcement learning (RL) improves the reasoning capabilities of large language models (LLMs), GRPO-style methods can suffer from a positive-credit contamination failure mode. Specifically, uniform credit assignment can reinforce locally erroneous tail tokens alongside useful tokens within successful trajectories. To address this problem, we propose Tail-Aware Credit calibratiOn (TACO), which calibrates the positive credit assigned to such tokens without token-level error labels. To this end, TACO lets the policy serve as its own reference, assessing each sampled token's rarity relative to the policy's local expectations to compute a tail-risk score. Using this score as a proxy for potential local errors, TACO softly reduces positive credit for high tail-risk tokens while retaining nonzero weights, limiting their reinforcement without eliminating opportunities to learn useful rare patterns. Experiments across four LLMs and eight reasoning benchmarks show that TACO consistently improves over GRPO-style baselines. Notably, TACO also improves training stability, supporting sustained performance gains in long-horizon RL. The source code is available at: https://github.com/xiuyilou/TACO.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑