超越均匀信用：为策略优化的因果信用分配
Beyond Uniform Credit: Causal Credit Assignment for Policy Optimization
AI总结：
本文提出反事实重要性加权方法，通过因果信用分配提升语言模型推理性能，无需辅助模型或外部标注，实验验证其在GSM8K上的有效性。
摘要：
语言模型推理的策略梯度方法，如GRPO和DAPO，对所有生成的token分配均匀的信用——填充短语“让我想想”获得的梯度更新与关键计算“23 + 45 = 68”相同。我们提出反事实重要性加权：屏蔽推理跨度，测量答案概率的下降，并在策略梯度更新中相应地提高token的权重。我们的方法不需要辅助模型或外部标注，重要性直接从策略模型自身的概率变化中估计。在GSM8K上跨三种模型（涵盖Qwen和Llama家族）的实验表明，与均匀基线相比，我们的方法在性能上一致提升，并且更快收敛到等效准确性。反转重要性信号会损害性能，证实我们捕捉到了真实的因果结构而不是噪声。分析显示，该方法正确地优先考虑计算步骤而非支撑文本。我们视这些发现为进一步研究的基础，而不是完整的解决方案。
英文摘要：
Policy gradient methods for language model reasoning, such as GRPO and DAPO, assign uniform credit to all generated tokens - the filler phrase "Let me think" receives the same gradient update as the critical calculation "23 + 45 = 68." We propose counterfactual importance weighting: mask reasoning spans, measure the drop in answer probability, and upweight tokens accordingly during policy gradient updates. Our method requires no auxiliary models or external annotation, instead importance is estimated directly from the policy model's own probability shifts. Experiments on GSM8K across three models spanning the Qwen and Llama families demonstrate consistent improvements over uniform baselines and faster convergence to equivalent accuracy. Inverting the importance signal hurts performance, confirming we capture genuine causal structure rather than noise. Analysis shows the method correctly prioritizes calculation steps over scaffolding text. We view these findings as establishing counterfactual importance weighting as a foundation for further research rather than a complete solution.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
