arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2603.21872cs.CVcs.AI
收藏

面向流形的探索用于视频生成的强化学习

Manifold-Aware Exploration for Reinforcement Learning in Video Generation

Tencent Hunyuan(腾讯文脉)

Mingzhe Zheng, Weijie Kong, Yue Wu, Dengyang Jiang, Yue Ma, Xuanhua He, Bin Lin, Kaixiong Gong, Zhao Zhong, Liefeng Bo, Qifeng Chen, Harry Yang

AI总结:

本文提出SAGE-GRPO方法,通过在微和宏观层面约束探索,确保在视频数据流形附近进行,提升视频生成的质量和奖励估计的可靠性。

摘要:

Group Relative Policy Optimization (GRPO)方法在视频生成中的应用,如FlowGRPO,相较于语言模型和图像模型仍不够可靠。这一差距源于视频生成具有复杂的解决方案空间,且用于探索的ODE到SDE转换会注入过多噪声,降低rollout质量并使奖励估计不可靠,从而影响训练后对齐。为此,本文将预训练模型视为定义有效视频数据流形,并将核心问题转化为约束探索在该流形附近,以保持rollout质量和奖励估计的可靠性。本文提出SAGE-GRPO(通过探索实现稳定对齐),在微观层面推导出具有对数曲率修正的精确流形感知SDE,并引入梯度范数均衡器以稳定采样和更新。在宏观层面,使用双信任区域和周期性移动锚点及分步约束,使信任区域跟踪更接近流形的检查点并限制长周期漂移。在HunyuanVideo1.5上评估SAGE-GRPO,使用原始VideoAlign作为奖励模型,并在VQ、MQ、TA和视觉指标(CLIPScore、PickScore)上观察到比先前方法一致的提升,证明在奖励最大化和整体视频质量上表现更优。代码和视觉画册可在https://dungeonmassster.github.io/SAGE-GRPO-Page/上获得。

英文摘要:

Group Relative Policy Optimization (GRPO) methods for video generation like FlowGRPO remain far less reliable than their counterparts for language models and images. This gap arises because video generation has a complex solution space, and the ODE-to-SDE conversion used for exploration can inject excess noise, lowering rollout quality and making reward estimates less reliable, which destabilizes post-training alignment. To address this problem, we view the pre-trained model as defining a valid video data manifold and formulate the core problem as constraining exploration within the vicinity of this manifold, ensuring that rollout quality is preserved and reward estimates remain reliable. We propose SAGE-GRPO (Stable Alignment via Exploration), which applies constraints at both micro and macro levels. At the micro level, we derive a precise manifold-aware SDE with a logarithmic curvature correction and introduce a gradient norm equalizer to stabilize sampling and updates across timesteps. At the macro level, we use a dual trust region with a periodic moving anchor and stepwise constraints so that the trust region tracks checkpoints that are closer to the manifold and limits long-horizon drift. We evaluate SAGE-GRPO on HunyuanVideo1.5 using the original VideoAlign as the reward model and observe consistent gains over previous methods in VQ, MQ, TA, and visual metrics (CLIPScore, PickScore), demonstrating superior performance in both reward maximization and overall video quality. The code and visual gallery are available at https://dungeonmassster.github.io/SAGE-GRPO-Page/.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑