arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2605.15980cs.CV
收藏

Flash-GRPO:通过单步策略优化实现视频扩散的高效对齐

Flash-GRPO: Efficient Alignment for Video Diffusion via One-Step Policy Optimization

Zhejiang University(浙江大学);Joy Future Academy(京东探索研究院);Independent Researcher(独立研究员);Tsinghua University(清华大学)

Xiaoxuan He, Siming Fu, Zeyue Xue, Weijie Wang, Ruizhe He, Yuming Li, Dacheng Yin, Shuai Dong, Haoyang Huang, Hongfa Wang, Nan Duan, Bohan Zhuang

AI总结:

提出Flash-GRPO单步训练框架,通过等时分组和时间梯度校正解决计算瓶颈,在低计算预算下实现优于全轨迹训练的对齐质量和训练效率。

摘要:

群体相对策略优化已成为将视频扩散模型与人类偏好对齐的关键,但面临一个关键的计算瓶颈:训练一个14B参数的模型通常每个实验需要数百个GPU天。现有的效率方法通过滑动窗口子采样训练时间步来降低成本,但从根本上损害了优化,表现出严重的不稳定性,并且无法达到完整的轨迹性能。我们提出了Flash-GRPO,一个单步训练框架,在低计算预算下在对齐质量上优于全轨迹训练,同时大幅提高了训练效率。Flash-GRPO解决了两个关键挑战:等时分组通过强制提示级别的时间一致性消除了时间步混淆的方差,将策略性能与时间步难度解耦;时间梯度校正中和了导致不同时间步梯度幅度极不一致的时间依赖缩放因子。在1.3B到14B参数模型上的实验验证了Flash-GRPO的有效性,展示了显著的训练加速,同时保持了一致的稳定性和最先进的对齐质量。

英文摘要:

Group Relative Policy Optimization has emerged as essential for aligning video diffusion models with human preferences, but faces a critical computational bottleneck: training a 14B parametered model typically demands hundreds of GPU days per experiment. Existing efficiency methods reduce costs through sliding window subsampling training timesteps, but fundamentally compromise optimization, exhibiting severe instability and failing to reach full trajectory performance. We present Flash-GRPO, a single-step training framework that outperforms full trajectory training in alignment quality under low computational budgets while substantially improving training efficiency. Flash-GRPO addresses two critical challenges: iso-temporal grouping eliminates timestep-confounded variance by enforcing prompt-wise temporal consistency, decoupling policy performance from timestep difficulty; temporal gradient rectification neutralizes the time-dependent scaling factor that causes vastly inconsistent gradient magnitudes across timesteps. Experiments on 1.3B to 14B parameter models validate Flash-GRPO's effectiveness, demonstrating substantial training acceleration with consistent stability and state-of-the-art alignment quality.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑