定位关键步骤：面向可验证强化学习的基于归因的信用分配
Pinpointing crucial steps: Attribution-based Credit Assignment for Verifiable Reinforcement Learning
AI总结：
针对RLVR中探索与利用失衡导致的信用分配不准和熵崩溃问题，提出分阶段框架ACPO，通过动态调节策略熵和量化推理步骤层级贡献，显著提升了LLM推理性能。
摘要：
虽然可验证奖励的强化学习（RLVR）提升了LLM的复杂推理能力，但现有方法难以平衡探索与利用。这导致了中间步骤信用分配不准确和熵过早崩溃等关键问题，限制了模型性能。为此，我们提出基于归因的策略优化贡献（ACPO），这是一个结合难度感知课程的分阶段框架。ACPO通过轨迹语义分割和基于归因的表示来动态调节策略熵以促进探索，从而缓解熵崩溃。同时，它利用分解的奖励系统精确量化每个推理步骤的层级贡献，确保准确的信用分配，从而增强利用。在AIME、MATH和AMC等基准上的实验表明，ACPO显著优于现有最先进方法。
英文摘要：
While Reinforcement Learning with Verifiable Rewards (RLVR) enhances complex reasoning in LLMs, current methods struggle to balance exploration and exploitation. This leads to critical issues like inaccurate credit assignment for intermediate steps and premature entropy collapse, limiting model performance. To address this, we introduce Attribution-based Contribution to Policy Optimization (ACPO), a phased framework that incorporates a difficulty-aware curriculum. ACPO improves exploration by using trajectory semantic segmentation and an attribution-based representation to dynamically regulate policy entropy, thus mitigating its collapse. Concurrently, it enhances exploitation with a factorized reward system that precisely quantifies the hierarchical contribution of each reasoning step, ensuring accurate credit assignment. Extensive experiments on challenging benchmarks, including AIME, MATH, and AMC, demonstrate that ACPO significantly outperforms existing state-of-the-art approaches.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
