MCPO：面向大推理模型的 mastery-consolidated 策略优化
MCPO: Mastery-Consolidated Policy Optimization for Large Reasoning Models
AI总结：
本文提出 MCPO 算法，通过引入 hinge-KL 正则化和优先机制，解决高准确度提示下的策略漂移和巩固问题，提升大模型推理性能。
摘要：
本文提出MCPO算法，通过引入hinge-KL正则化和优先机制，解决高准确度提示下的策略漂移和巩固问题，提升大模型推理性能。
英文摘要：
Reinforcement Learning with Verifiable Rewards (RLVR) has emerged as a promising approach to improve the reasoning abilities of Large Language Models (LLMs). Among RLVR algorithms, Group Relative Policy Optimization (GRPO) and its variants have demonstrated strong performance and high training efficiency. However, GRPO-style objectives exhibit two issues on high accuracy prompts including mastered prompts (rollout accuracy =1) and majority-correct prompts (rollout accuracy in (0.5,1)). For mastered prompts, group-relative advantages vanish, yielding no training signal and unconstrained policy drift that can cause forgetting. For majority-correct prompts, the induced query weight shrinks as accuracy increases, weakening consolidation from partial correctness to mastery. To alleviate this, we propose Mastery-Consolidated Policy Optimization (MCPO), which introduces (i) a hinge-KL regularizer applied exclusively to mastered prompts to bound harmful policy drift between successive gradient steps, and (ii) a weighting mechanism that prioritizes majority-correct prompts to better allocate optimization effort. Extensive experiments across three mathematical benchmarks demonstrate that MCPO consistently improves pass@1 performance. Counter-intuitively, rather than restricting exploration, MCPO boosts pass@k metrics, indicating that mastery consolidation further catalyzes solution diversity.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
