arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2601.05729cs.CV
收藏

TAGRPO: 通过直接轨迹对齐提升图像到视频生成中的GRPO

TAGRPO: Boosting GRPO on Image-to-Video Generation with Direct Trajectory Alignment

The University of Hong Kong(香港大学);Tencent Hunyuan(腾讯文心)

Jin Wang, Jianxiang Lu, Guangzheng Xu, Comi Chen, Haoyu Yang, Linqing Wang, Peng Chen, Mingtao Chen, Zhichao Hu, Longhuang Wu, Shuai Shao, Qinglin Lu, Ping Luo

AI总结:

针对图像到视频生成中GRPO优化效果不佳的问题,提出基于对比学习的TAGRPO框架,通过中间潜变量对齐高奖励轨迹并远离低奖励轨迹,结合记忆库提升多样性,显著优于DanceGRPO。

摘要:

近期研究表明,将组相对策略优化(GRPO)集成到流匹配模型中,特别是在文本到图像和文本到视频生成中,具有显著效果。然而,我们发现将这些技术直接应用于图像到视频(I2V)模型往往无法带来一致的奖励提升。为解决这一局限,我们提出了TAGRPO,一个受对比学习启发的鲁棒后训练框架,适用于I2V模型。我们的方法基于以下观察:从相同初始噪声生成的rollout视频为优化提供了更优的指导。基于这一洞察,我们提出了一种应用于中间潜变量的新型GRPO损失,鼓励直接对齐高奖励轨迹,同时最大化与低奖励轨迹的距离。此外,我们引入了一个用于rollout视频的记忆库,以增强多样性并降低计算开销。尽管方法简单,TAGRPO在I2V生成中相比DanceGRPO取得了显著改进。相关成果将在 https://tagrpo.github.io/ 更新。

英文摘要:

Recent studies have demonstrated the efficacy of integrating Group Relative Policy Optimization (GRPO) into flow matching models, particularly for text-to-image and text-to-video generation. However, we find that directly applying these techniques to image-to-video (I2V) models often fails to yield consistent reward improvements. To address this limitation, we present TAGRPO, a robust post-training framework for I2V models inspired by contrastive learning. Our approach is grounded in the observation that rollout videos generated from identical initial noise provide superior guidance for optimization. Leveraging this insight, we propose a novel GRPO loss applied to intermediate latents, encouraging direct alignment with high-reward trajectories while maximizing distance from low-reward counterparts. Furthermore, we introduce a memory bank for rollout videos to enhance diversity and reduce computational overhead. Despite its simplicity, TAGRPO achieves significant improvements over DanceGRPO in I2V generation. The deliverables will be updated at https://tagrpo.github.io/ .

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑