AnyV2V: 面向任意视频到视频编辑任务的免调优框架
AnyV2V: A Tuning-Free Framework For Any Video-to-Video Editing Tasks
AI总结：
针对现有视频编辑模型质量不足、控制受限、依赖文本引导且需微调的问题，提出免调优框架AnyV2V，通过首帧编辑加时序特征注入的图像到视频生成实现多类编辑任务，在人工评估中视觉一致性表现更优。
摘要：
在使用生成模型的数字内容创作这一动态领域中，当前最先进的视频编辑模型仍无法达到用户期望的质量与控制水平。以往的视频编辑研究要么以零样本方式从基于图像的生成模型扩展而来，要么需要大量微调，这会阻碍流畅视频编辑效果的产出。此外，这些方法常依赖文本输入作为编辑引导，导致歧义并限制了可执行的编辑类型。针对这些挑战，我们提出AnyV2V，一种全新的免调优范式，将视频编辑简化为两个核心步骤：（1）使用现成的图像编辑模型修改首帧；（2）利用现有的图像到视频生成模型，通过时序特征注入生成编辑后的视频。AnyV2V可借助任意现有图像编辑工具，支持广泛的视频编辑任务，包括基于提示的编辑、基于参考的风格迁移、主体驱动编辑以及身份操控，这些都是以往方法无法实现的。AnyV2V还支持任意视频长度。我们的评估显示，AnyV2V取得了与其他基线方法相当的CLIP得分。此外，AnyV2V在人工评估中显著优于这些基线，在所有编辑任务中生成高质量编辑效果的同时，在与源视频的视觉一致性方面展现出显著提升。
英文摘要：
In the dynamic field of digital content creation using generative models, state-of-the-art video editing models still do not offer the level of quality and control that users desire. Previous works on video editing either extended from image-based generative models in a zero-shot manner or necessitated extensive fine-tuning, which can hinder the production of fluid video edits. Furthermore, these methods frequently rely on textual input as the editing guidance, leading to ambiguities and limiting the types of edits they can perform. Recognizing these challenges, we introduce AnyV2V, a novel tuning-free paradigm designed to simplify video editing into two primary steps: (1) employing an off-the-shelf image editing model to modify the first frame, (2) utilizing an existing image-to-video generation model to generate the edited video through temporal feature injection. AnyV2V can leverage any existing image editing tools to support an extensive array of video editing tasks, including prompt-based editing, reference-based style transfer, subject-driven editing, and identity manipulation, which were unattainable by previous methods. AnyV2V can also support any video length. Our evaluation shows that AnyV2V achieved CLIP-scores comparable to other baseline methods. Furthermore, AnyV2V significantly outperformed these baselines in human evaluations, demonstrating notable improvements in visual consistency with the source video while producing high-quality edits across all editing tasks.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
