VIVA: 基于VLM的指令式视频编辑与奖励优化
VIVA: VLM-Guided Instruction-Based Video Editing with Reward Optimization
AI总结：
VIVA通过VLM引导编码和奖励优化，实现更高效、高质量的指令式视频编辑，提升对复杂指令的泛化能力。
摘要：
基于指令的视频编辑旨在根据自然语言指令修改输入视频，同时保持内容保真度和时间一致性。然而，现有扩散模型方法通常是在简单编辑操作的配对数据上训练，这从根本上限制了它们对多样且复杂的现实世界指令的泛化能力。为了解决这一泛化差距，我们提出了VIVA，一个可扩展的指令式视频编辑框架，该框架利用VLM引导的编码和奖励优化。首先，我们引入了一个基于VLM的指导者，将文本指令、源视频的第一帧和可选参考图像编码为视觉引导的指令表示，为扩散变换器主干提供细粒度的空间和语义上下文。其次，我们提出了一种后训练阶段，Edit-GRPO，它将Group Relative Policy Optimization适应到视频编辑领域，直接优化模型以实现符合指令、保持内容和具有审美吸引力的编辑，使用相对奖励。此外，我们提出了一种数据构建管道，旨在合成生成多样、高保真的基本编辑操作视频-指令配对数据。广泛的实验表明，VIVA在指令遵循、泛化和编辑质量方面均优于最先进的方法。网站：https://viva-paper.github.io
英文摘要：
Instruction-based video editing aims to modify an input video according to a natural-language instruction while preserving content fidelity and temporal coherence. However, existing diffusion-based approaches are often trained on paired data of simple editing operations, which fundamentally limits their ability to generalize to diverse and complex, real-world instructions. To address this generalization gap, we propose VIVA, a scalable framework for instruction-based video editing that leverages VLM-guided encoding and reward optimization. First, we introduce a VLM-based instructor that encodes the textual instruction, the first frame of the source video, and an optional reference image into visually-grounded instruction representations, providing fine-grained spatial and semantic context for the diffusion transformer backbone. Second, we propose a post-training stage, Edit-GRPO, which adapts Group Relative Policy Optimization to the domain of video editing, directly optimizing the model for instruction-faithful, content-preserving, and aesthetically pleasing edits using relative rewards. Furthermore, we propose a data construction pipeline designed to synthetically generate diverse, high-fidelity paired video-instruction data of basic editing operations. Extensive experiments show that VIVA achieves superior instruction following, generalization, and editing quality over state-of-the-art methods. Website: https://viva-paper.github.io
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
