所见即编辑：基于掩码运动建模的图像引导视频编辑
Edit as You See: Image-guided Video Editing via Masked Motion Modeling
AI总结：
针对图像引导视频编辑，本文提出 IVEDiff，通过可学习运动模块、掩码运动建模微调和光流引导运动参考网络保持时序一致性，并构建基准验证其高质量鲁棒编辑效果。
摘要：
扩散模型的最新进展显著推动了文本引导的视频编辑。然而，关于图像引导视频编辑的研究相对匮乏；该方法允许用户仅通过在初始帧中标定目标对象并提供一张 RGB 图像作为参考来编辑视频，而无需依赖文本提示。本文提出一种新颖的图像引导视频编辑扩散模型，命名为 IVEDiff，用于图像引导视频编辑。IVEDiff 构建于图像编辑模型之上，并配备可学习的运动模块，以维持编辑后视频的时间一致性。受自监督学习思想启发，我们引入掩码运动建模微调策略，使运动模块能够捕捉帧间运动动态，同时保留基础图像编辑模型对帧内语义相关性建模的能力。此外，我们提出光流引导的运动参考网络，以确保编辑后视频帧之间信息的准确传播，缓解无效信息的误导影响。我们还构建了一个基准以促进后续研究。综合实验表明，本方法能够生成时间上平滑的编辑视频，并能高质量、鲁棒地处理各类编辑对象。
英文摘要：
Recent advancements in diffusion models have significantly facilitated text-guided video editing. However, there is a relative scarcity of research on image-guided video editing, a method that empowers users to edit videos by merely indicating a target object in the initial frame and providing an RGB image as reference, without relying on the text prompts. In this paper, we propose a novel Image-guided Video Editing Diffusion model, termed IVEDiff for the image-guided video editing. IVEDiff is built on top of image editing models, and is equipped with learnable motion modules to maintain the temporal consistency of edited video. Inspired by self-supervised learning concepts, we introduce a masked motion modeling fine-tuning strategy that empowers the motion module's capabilities for capturing inter-frame motion dynamics, while preserving the capabilities for intra-frame semantic correlations modeling of the base image editing model. Moreover, an optical-flow-guided motion reference network is proposed to ensure the accurate propagation of information between edited video frames, alleviating the misleading effects of invalid information. We also construct a benchmark to facilitate further research. The comprehensive experiments demonstrate that our method is able to generate temporally smooth edited videos while robustly dealing with various editing objects with high quality.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
