Probing RLVR training instability through the lens of objective-level hacking
School of Physics, Peking University(北京大学物理学院);Tongyi Lab(通义实验室);Alibaba Group(阿里巴巴集团);Kavli Institute for Astronomy and Astrophysics, Peking University(北京大学天文与天体物理研究院);National Astronomical Observatories, Chinese Academy of Sciences(中国科学院国家天文台)
Yiming Dong, Kun Fu, Haoyu Li, Xinyuan Zhu, Yurou Liu, Lijing Shao, Jieping Ye, Zheng Wang
Prolonged reinforcement learning with verifiable rewards (RLVR) has been shown to drive continuous improvements in the reasoning capabilities of large language models, but the training is often prone to instabilities, especially in Mixture-of-Experts (MoE) architectures. Training instability severely undermines model capability improvement, yet its underlying causes and mechanisms remain poorly understood. In this work, we introduce a principled framework for understanding RLVR instability through the lens of objective-level hacking. Unlike reward hacking, which arises from exploitable verifiers, objective-level hacking emerges from token-level credit misalignment and is manifested as system-level spurious signals in the optimization objective. Grounded in our framework, together with extensive experiments on a 30B MoE model, we trace the origin and formalize the mechanism behind a key pathological training dynamic in MoE models: the abnormal growth of the training-inference discrepancy, a phenomenon widely associated with instability but previously lacking a mechanistic explanation. These findings provide a concrete and causal account of the training dynamics underlying instabilities in MoE models, offering guidance for the design of stable RLVR algorithms.
Understanding Diversity Collapse in RLVR via the Lens of Overtraining
Sydney AI Centre, The University of Sydney(悉尼大学悉尼人工智能中心);Southeast University(东南大学);Microsoft(微软);Data61, CSIRO(澳大利亚联邦科学与工业研究组织Data61);Chongqing University(重庆大学);Nanyang Technological University(南洋理工大学)
ContextRL: Enhancing MLLM's Knowledge Discovery Efficiency with Context-Augmented RL
Tsinghua Shenzhen International Graduate School, Tsinghua University(清华大学深圳国际研究生院);Harbin Institute of Technology, Shenzhen(哈尔滨工业大学(深圳));Chinese Academy of Sciences(中国科学院);Tsinghua University(清华大学)