VEGGIE：基于接地生成的教学编辑与视频概念推理
VEGGIE: Instructional Editing and Reasoning Video Concepts with Grounded Generation
AI总结：
VEGGIE提出端到端框架，统一教学视频编辑、接地与推理，通过MLLM和扩散模型结合课程学习及数据合成，在多项任务上超越基线。
摘要：
最近的视频扩散模型增强了视频编辑能力，但在统一框架内处理教学编辑和多样化任务（如添加、移除、更改）仍然具有挑战性。在本文中，我们介绍了VEGGIE，一种基于指令接地生成的视频编辑器，这是一个简单的端到端框架，统一了基于多样化用户指令的视频概念编辑、接地和推理。具体来说，给定一个视频和文本查询，VEGGIE首先利用多模态大语言模型（MLLM）解释指令中的用户意图，并将其接地到视频上下文中，生成针对帧的接地任务查询，用于像素空间的响应。然后，扩散模型渲染这些计划并生成与用户意图一致的编辑视频。为了支持多样化任务和复杂指令，我们采用了课程学习策略：首先使用大规模教学图像编辑数据对齐MLLM和视频扩散模型，然后在高质量多任务视频数据上进行端到端微调。此外，我们引入了一种新颖的数据合成流程，用于生成成对的教学视频编辑数据以进行模型训练。它通过利用图像到视频（Image-to-Video）模型注入动态，将静态图像数据转换为多样化、高质量的视频编辑样本。VEGGIE在教学视频编辑中展现了强大的性能，涵盖不同的编辑技能，作为多功能模型优于最佳教学基线，而其他模型在多任务处理上表现不佳。VEGGIE还在视频对象接地和推理分割方面表现出色，而其他基线则失败。我们进一步揭示了多个任务如何相互促进，并突出了有前景的应用，如零样本多模态教学和上下文内视频编辑。
英文摘要：
Recent video diffusion models have enhanced video editing, but it remains challenging to handle instructional editing and diverse tasks (e.g., adding, removing, changing) within a unified framework. In this paper, we introduce VEGGIE, a Video Editor with Grounded Generation from Instructions, a simple end-to-end framework that unifies video concept editing, grounding, and reasoning based on diverse user instructions. Specifically, given a video and text query, VEGGIE first utilizes an MLLM to interpret user intentions in instructions and ground them to the video contexts, generating frame-specific grounded task queries for pixel-space responses. A diffusion model then renders these plans and generates edited videos that align with user intent. To support diverse tasks and complex instructions, we employ a curriculum learning strategy: first aligning the MLLM and video diffusion model with large-scale instructional image editing data, followed by end-to-end fine-tuning on high-quality multitask video data. Additionally, we introduce a novel data synthesis pipeline to generate paired instructional video editing data for model training. It transforms static image data into diverse, high-quality video editing samples by leveraging Image-to-Video models to inject dynamics. VEGGIE shows strong performance in instructional video editing with different editing skills, outperforming the best instructional baseline as a versatile model, while other models struggle with multi-tasking. VEGGIE also excels in video object grounding and reasoning segmentation, where other baselines fail. We further reveal how the multiple tasks help each other and highlight promising applications like zero-shot multimodal instructional and in-context video editing.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
