分步信用分配用于流匹配模型中的GRPO
Stepwise Credit Assignment for GRPO on Flow-Matching Models
AI总结：
本文提出分步流GRPO，通过分步奖励改进提升样本效率和收敛速度，引入DDIM启发的SDE提升奖励质量。
摘要：
Flow-GRPO成功将强化学习应用于流模型，但使用统一的信用分配方式。这忽略了扩散生成的时间结构：早期步骤决定组成和内容（低频结构），而后期步骤解决细节和纹理（高频细节）。此外，仅基于最终图像分配统一信用可能无意中奖励次优中间步骤，尤其是在后期扩散轨迹中纠正错误时。我们提出分步流GRPO，通过每个步骤的奖励改进分配信用。通过利用Tweedie公式获得中间奖励估计，并引入基于收益的优势，我们的方法实现了更高的样本效率和更快的收敛速度。我们还引入了DDIM启发的SDE，以提高奖励质量同时保持对策略梯度的随机性。
英文摘要：
Flow-GRPO successfully applies reinforcement learning to flow models, but uses uniform credit assignment across all steps. This ignores the temporal structure of diffusion generation: early steps determine composition and content (low-frequency structure), while late steps resolve details and textures (high-frequency details). Moreover, assigning uniform credit based solely on the final image can inadvertently reward suboptimal intermediate steps, especially when errors are corrected later in the diffusion trajectory. We propose Stepwise-Flow-GRPO, which assigns credit based on each step's reward improvement. By leveraging Tweedie's formula to obtain intermediate reward estimates and introducing gain-based advantages, our method achieves superior sample efficiency and faster convergence. We also introduce a DDIM-inspired SDE that improves reward quality while preserving stochasticity for policy gradients.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
