arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2610.26022cs.CVcs.AI
收藏

FrameGRPO:通过多帧展开和时间信用激励视频推理

FrameGRPO: Incentivizing Video Reasoning via Multi-Frame Rollouts and Temporal Credit

Shanghai Jiao Tong University(上海交通大学);Shanghai Innovation Institute(上海创新研究院);Tencent(腾讯)

Jiaxuan Sun, Biao Wang, Shuai Li

AI总结:

针对标准视频GRPO的同质化奖励问题,提出无需辅助网络或架构修改的FrameGRPO,通过多帧展开和门控时间效率信用提升视频推理性能,在多类基准上取得稳健改进。

摘要:

带有可验证奖励的强化学习,尤其是组相对策略优化(GRPO),已提升了多模态大语言模型的视频推理能力。然而,标准视频GRPO仅改变语言轨迹,同时将组内每次展开都基于相同采样帧进行条件设置,当共享视觉证据不足或存在干扰时,会产生同质化奖励并削弱组相对学习。我们提出FrameGRPO,这是一种简单有效的强化学习框架,既不需要辅助网络,也无需修改模型架构。具体而言,FrameGRPO结合了两个轻量级组件:(1)多帧展开,仅对单个最大帧解码进行均匀时间子采样,以构建多样化视觉证据和推理轨迹的对齐视图;(2)门控时间效率信用,当固定全比例参考子集内的所有展开均失败时,有选择地强化正确的低比例轨迹,而不对更少帧施加无条件偏好。在多种视频理解与推理基准上开展的大量实验表明,其相比标准GRPO具有稳健的整体改进,凸显时间视图多样化是视频多模态大语言模型中探索与信用分配的有效机制。

英文摘要:

Reinforcement learning with verifiable rewards, particularly Group Relative Policy Optimization (GRPO), has improved video reasoning in multimodal large language models. However, standard video GRPO varies only linguistic trajectories while conditioning every rollout in a group on the same sampled frames, which can produce homogeneous rewards and weaken group-relative learning when the shared visual evidence is insufficient or distracting. We propose \textbf{FrameGRPO}, a simple and effective reinforcement-learning framework that requires neither auxiliary networks nor changes to the model architecture. Specifically, FrameGRPO combines two lightweight components: (1) \textbf{Multi-Frame Rollouts}, which use only uniform temporal subsampling of a single maximal-frame decode to construct aligned views that diversify visual evidence and reasoning trajectories; and (2) \textbf{Gated Temporal Efficiency Credit}, which selectively reinforces a correct lower-ratio trajectory when all rollouts in a fixed full-ratio reference subset fail, without imposing an unconditional preference for fewer frames. Extensive experiments across diverse video understanding and reasoning benchmarks demonstrate robust overall improvements over standard GRPO, highlighting temporal-view diversification as an effective mechanism for exploration and credit assignment in video MLLMs.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑