arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2604.04142cs.CV
收藏

OP-GRPO:用于流匹配模型的高效非策略GRPO

OP-GRPO: Efficient Off-Policy GRPO for Flow-Matching Models

College of Control Science and Engineering, Zhejiang University(浙江大学控制科学与工程学院);Central Research Institute, Huawei(华为中央研究院)

Liyu Zhang, Kehan Li, Tingrui Han, Tao Zhao, Yuxuan Sheng, Shibo He, Chao Li

AI总结:

OP-GRPO通过高效非策略训练方法提升流匹配模型生成质量,通过高质轨迹选择与重要采样修正,实现训练效率提升34.2%。

摘要:

在GRPO后训练中,流匹配模型的生成质量显著提升,但其策略训练范式导致样本效率低下。为解决此问题,我们提出OP-GRPO,首个针对流匹配模型的非策略GRPO框架。首先,我们主动选择高质量轨迹并将其适配到回放缓冲区以供后续训练迭代使用。其次,为缓解非策略样本引入的分布偏移,我们提出序列级重要性采样修正,保留GRPO的裁剪机制完整性并确保稳定策略更新。第三,我们理论和实验证明,后期去噪步骤导致非策略比率病态,通过在后期步骤截断轨迹以缓解此问题。在图像和视频生成基准测试中,OP-GRPO仅需平均34.2%的训练步骤即可达到与Flow-GRPO相当或更优的性能,显著提升训练效率并保持生成质量。

英文摘要:

Post training via GRPO has demonstrated remarkable effectiveness in improving the generation quality of flow-matching models. However, GRPO suffers from inherently low sample efficiency due to its on-policy training paradigm. To address this limitation, we present OP-GRPO, the first Off-Policy GRPO framework tailored for flow-matching models. First, we actively select high-quality trajectories and adaptively incorporate them into a replay buffer for reuse in subsequent training iterations. Second, to mitigate the distribution shift introduced by off-policy samples, we propose a sequence-level importance sampling correction that preserves the integrity of GRPO's clipping mechanism while ensuring stable policy updates. Third, we theoretically and empirically show that late denoising steps yield ill-conditioned off-policy ratios, and mitigate this by truncating trajectories at late steps. Across image and video generation benchmarks, OP-GRPO achieves comparable or superior performance to Flow-GRPO with only 34.2% of the training steps on average, yielding substantial gains in training efficiency while maintaining generation quality.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑