基于锚定策略优化:通过支持约束修正缓解探索崩溃
Anchored Policy Optimization: Mitigating Exploration Collapse Via Support-Constrained Rectification
AI总结:
本文提出锚定策略优化方法,通过支持覆盖机制缓解强化学习中的探索崩溃问题,提升模型在准确性与多样性间的性能平衡。
摘要:
可验证奖励的强化学习(RLVR)日益被视为一种树剪枝机制。然而,我们发现一种系统性病理现象,称为递归空间收缩(RSC),即由正向锐化和负向挤压的联合动态驱动的不可逆崩溃,其中有效替代方案的采样概率消失。尽管Kullback-Leibler(KL)正则化旨在缓解这一问题,但其施加了刚性的形状匹配约束,迫使策略模仿参考模型的完整密度,从而与正确性所需的锐化产生梯度冲突。我们提出锚定策略优化(APO),从全局形状匹配转向支持覆盖。通过基于参考模型高置信度支持定义安全流形,APO允许为效率进行激进的锐化,同时在错误纠正期间选择性地调用恢复力以防止崩溃。我们理论推导出APO作为一种梯度对齐机制,以最大化支持覆盖,使弹性恢复能够重新填充有效分支。在数学基准上的经验评估表明,APO打破了准确性-多样性权衡,显著提高了Pass@1的同时恢复了标准策略梯度方法通常丢失的Pass@K多样性。
英文摘要:
Reinforcement Learning with Verifiable Rewards (RLVR) is increasingly viewed as a tree pruning mechanism. However, we identify a systemic pathology termed Recursive Space Contraction (RSC), an irreversible collapse driven by the combined dynamics of positive sharpening and negative squeezing, where the sampling probability of valid alternatives vanishes. While Kullback-Leibler (KL) regularization aims to mitigate this, it imposes a rigid Shape Matching constraint that forces the policy to mimic the reference model's full density, creating a gradient conflict with the sharpening required for correctness. We propose Anchored Policy Optimization (APO), shifting the paradigm from global Shape Matching to Support Coverage. By defining a Safe Manifold based on the reference model's high-confidence support, APO permits aggressive sharpening for efficiency while selectively invoking a restorative force during error correction to prevent collapse. We theoretically derive that APO serves as a gradient-aligned mechanism to maximize support coverage, enabling an Elastic Recovery that re-inflates valid branches. Empirical evaluations on mathematical benchmarks demonstrate that APO breaks the accuracy-diversity trade-off, significantly improving Pass@1 while restoring the Pass@K diversity typically lost by standard policy gradient methods.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。