CAPO：通过生成式信用分配增强 LLM 推理能力
CAPO: Towards Enhancing LLM Reasoning through Generative Credit Assignment
AI总结：
提出 CAPO，利用通用 LLM 作为生成式过程奖励模型产生确定性 token 级信用，并通过投票增强鲁棒性，从而提升 LLM 数学及域外推理性能。
摘要：
带可验证奖励的强化学习（RLVR）通过基于规则的二元反馈提升了大型语言模型（LLM）的推理能力。然而，当前 RLVR 方法通常给每个 token 分配相同奖励。这种粗粒度反馈妨碍了精确的信用分配，使模型难以识别哪些推理步骤导致成功或失败，并常产生次优策略。PPO 等方法通过价值估计提供信用分配，但由于采样有限，会产生不准确且不可验证的信号。另一方面，使用过程奖励模型的方法可提供逐步奖励，却存在若干关键局限：它们需要高质量的过程监督标签，概率性奖励建模导致反馈不可靠，而且在在线强化学习（RL）中应用非常耗时。为克服这些局限，我们提出一种简单而高效的方法——信用分配策略优化（CAPO）。CAPO 不训练辅助模型，而是直接利用现成的通用 LLM 作为生成式过程奖励模型（LLM-as-GenPRM），仅依据步骤本身的正确性，通过一次生成产生所有逐步批评，从而提供确定性的 token 级信用，以改进最初被赋予相同规则奖励的 token。为进一步提升准确性和鲁棒性，我们采用随生成批评数量扩展的投票机制。在 Llama 和 Qwen 等多种骨干模型上的大量实验表明，CAPO 在四个具有挑战性的数学基准和三个域外基准上始终优于基于监督学习和基于 RL 的微调方法。进一步分析表明，CAPO 能够帮助模型促进对可导向正确答案的正确推理路径的学习。
英文摘要：
Reinforcement Learning with Verifiable Rewards (RLVR) has improved the reasoning abilities of Large Language Models (LLMs) by using rule-based binary feedback. However, current RLVR methods typically assign the same reward to every token. This coarse-grained feedback hampers precise credit assignment, making it hard for models to identify which reasoning steps lead to success or failure, and often results in suboptimal policies. Methods like PPO provide credit assignment by value estimation, but yield inaccurate and unverifiable signals due to limited sampling. On the other hand, methods using Process Reward Models can provide step-wise rewards but suffer from several key limitations: they require high-quality process supervision labels, the feedback is unreliable due to probabilistic reward modeling, and their application in online reinforcement learning (RL) is time-consuming. To overcome these limitations, we introduce a simple but efficient method-Credit Assignment Policy Optimization (CAPO). Instead of training auxiliary models, CAPO directly leverages an off-the-shelf, general-purpose LLM as a Generative Process Reward Model (LLM-as-GenPRM) to generate all step-wise critique by one pass only based on the correctness of the step itself, providing deterministic token-level credits to refine the tokens that were originally assigned identical rule-based rewards. To further enhance the accuracy and robustness, we employ voting mechanisms that scale with the number of generated critiques. Extensive experiments on various backbones like Llama and Qwen models show that CAPO consistently outperforms supervised learning-based and RL-based fine-tuning methods across four challenging mathematical benchmarks and three out-of-domain benchmarks. Further analysis shows that CAPO can help the model to foster the learning of correct reasoning pathways leading to correct answers.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
