论组相对策略优化的策略梯度基础：信用分配、梯度稀疏性与秩坍塌
On the Policy Gradient Foundations of Group Relative Policy Optimization: Credit Assignment, Gradient Sparsity, and Rank Collapse
AI总结：
本文从策略梯度定理推导GRPO，揭示其输出奖励下所有token获得相同优势导致信用分配失败，证明梯度稀疏性并经验证梯度矩阵有效秩约为2，形式化秩-2结构源于优势零和约束。
摘要：
组相对策略优化（GRPO）通过使用分组 rollout 的平均奖励作为基线，消除了 PPO 中的学习评论家。我们从策略梯度定理的第一性原理出发，对 GRPO 进行了严格推导，揭示了一个基本的信用分配失败：在仅输出奖励的情况下，rollout 中的每个 token 获得相同的优势，将 token 级别的信用坍缩为单个标量。我们证明这会导致梯度稀疏性随训练加剧，并通过 Nemotron-4B/GSM8K 上 GRPO 梯度的 SVD 分析经验性地表明，无论组大小 $R \in \{2, 4, 8\}$，梯度矩阵的有效秩约为 2。我们将此形式化为由优势的零和约束引起的内在秩-2 结构，并推导出 GRPO 基线最优的条件。我们的结果刻画了 GRPO 的简单性在理论上何时合理，并指出信用分配瓶颈是多步推理的关键限制。
英文摘要：
Group Relative Policy Optimization (GRPO) eliminates the learned critic in PPO by using the mean reward of grouped rollouts as a baseline. We provide a rigorous derivation of GRPO from first principles of the policy gradient theorem, revealing a fundamental credit assignment failure: under output-only reward, every token in a rollout receives identical advantage, collapsing token-level credit to a single scalar. We prove this induces gradient sparsity that intensifies over training, and demonstrate empirically via SVD analysis of GRPO gradients on Nemotron-4B/GSM8K that the gradient matrix has effective rank $\approx$ 2 regardless of group size $R \in \{2, 4, 8\}$. We formalize this as an intrinsic rank-2 structure arising from the zero-sum constraint on advantages and derive conditions under which GRPO's baseline is optimal. Our results characterize when GRPO's simplicity is theoretically justified and identify the credit assignment bottleneck as the key limitation for multi-step reasoning.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
