arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2602.01103cs.AI
收藏

通过目标层面黑客的视角探查RLVR训练不稳定性

Probing RLVR training instability through the lens of objective-level hacking

School of Physics, Peking University(北京大学物理学院);Tongyi Lab(通义实验室);Alibaba Group(阿里巴巴集团);Kavli Institute for Astronomy and Astrophysics, Peking University(北京大学天文与天体物理研究院);National Astronomical Observatories, Chinese Academy of Sciences(中国科学院国家天文台)

Yiming Dong, Kun Fu, Haoyu Li, Xinyuan Zhu, Yurou Liu, Lijing Shao, Jieping Ye, Zheng Wang

AI总结:

本文通过目标层面黑客视角揭示RLVR训练不稳定性的根源,分析MoE模型中训练-推理差异异常增长的机制,为设计稳定的RLVR算法提供指导。

摘要:

长时间的可验证奖励强化学习(RLVR)已被证明能持续提升大语言模型的推理能力,但训练过程常出现不稳定性,尤其在混合专家(MoE)架构中更为明显。训练不稳定性严重阻碍模型能力提升,但其根本原因和机制尚不明确。本文引入一个系统框架,通过目标层面黑客的视角理解RLVR的不稳定性。不同于奖励黑客,后者源于可利用的验证器,目标层面黑客则源于token级信用错配,并表现为优化目标中的系统级虚假信号。基于该框架,结合对30B MoE模型的广泛实验,我们追踪了MoE模型中关键病态训练动态的起源,并正式化其背后的机制:训练-推理差异的异常增长,这一现象广泛与不稳定性相关,但此前缺乏机制性解释。这些发现为MoE模型不稳定性背后的训练动态提供了具体且因果的解释,为设计稳定的RLVR算法提供了指导。

英文摘要:

Prolonged reinforcement learning with verifiable rewards (RLVR) has been shown to drive continuous improvements in the reasoning capabilities of large language models, but the training is often prone to instabilities, especially in Mixture-of-Experts (MoE) architectures. Training instability severely undermines model capability improvement, yet its underlying causes and mechanisms remain poorly understood. In this work, we introduce a principled framework for understanding RLVR instability through the lens of objective-level hacking. Unlike reward hacking, which arises from exploitable verifiers, objective-level hacking emerges from token-level credit misalignment and is manifested as system-level spurious signals in the optimization objective. Grounded in our framework, together with extensive experiments on a 30B MoE model, we trace the origin and formalize the mechanism behind a key pathological training dynamic in MoE models: the abnormal growth of the training-inference discrepancy, a phenomenon widely associated with instability but previously lacking a mechanistic explanation. These findings provide a concrete and causal account of the training dynamics underlying instabilities in MoE models, offering guidance for the design of stable RLVR algorithms.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑