DreamVE：统一的基于指令的图像与视频编辑
DreamVE: Unified Instruction-based Image and Video Editing
AI总结：
提出DreamVE，一个统一的基于指令的图像与视频编辑模型，采用两阶段训练策略和多样化数据合成流水线，基于SOTA T2V模型构建高效编辑框架，解决训练数据受限问题并提升编辑性能。
摘要：
基于指令的编辑因其简单高效的交互式编辑格式而具有巨大潜力。然而，基于指令的编辑（尤其是视频编辑）受限于有限的训练数据，阻碍了其实际应用。为此，我们提出了DreamVE，一个用于基于指令的图像和视频编辑的统一模型。具体而言，我们提出了一种两阶段训练策略：先进行图像编辑，再进行视频编辑。这带来了两大主要优势：（1）图像数据更易于规模化，且模型训练效率更高，为更快、更好的视频编辑训练提供了有用的先验知识。（2）统一图像和视频生成是自然的，且符合当前趋势。此外，我们提出了全面的训练数据合成流水线，包括基于拼贴和基于生成模型的数据合成。基于拼贴的数据合成将前景对象和背景相结合，以生成多样化的编辑数据，例如对象操作、背景更改和文本修改。它可以轻松生成数十亿个准确、一致、逼真且多样化的编辑对。我们在大量基于拼贴的数据上预训练DreamVE，以在关键编辑类型中实现强大性能，并增强泛化和迁移能力。然而，基于拼贴的数据缺乏某些属性编辑案例，导致性能相对下降。相比之下，基于生成模型的流水线虽然难以规模化，但在处理属性编辑案例时具有灵活性。因此，我们使用基于生成模型的数据进一步微调DreamVE。此外，我们为DreamVE设计了一个高效且强大的编辑框架。我们基于SOTA T2V模型构建，并使用token拼接与早期丢弃方法注入源图像引导，以确保强一致性和可编辑性。代码和模型将被发布。
英文摘要：
Instruction-based editing holds vast potential due to its simple and efficient interactive editing format. However, instruction-based editing, particularly for video, has been constrained by limited training data, hindering its practical application. To this end, we introduce DreamVE, a unified model for instruction-based image and video editing. Specifically, We propose a two-stage training strategy: first image editing, then video editing. This offers two main benefits: (1) Image data scales more easily, and models are more efficient to train, providing useful priors for faster and better video editing training. (2) Unifying image and video generation is natural and aligns with current trends. Moreover, we present comprehensive training data synthesis pipelines, including collage-based and generative model-based data synthesis. The collage-based data synthesis combines foreground objects and backgrounds to generate diverse editing data, such as object manipulation, background changes, and text modifications. It can easily generate billions of accurate, consistent, realistic, and diverse editing pairs. We pretrain DreamVE on extensive collage-based data to achieve strong performance in key editing types and enhance generalization and transfer capabilities. However, collage-based data lacks some attribute editing cases, leading to a relative drop in performance. In contrast, the generative model-based pipeline, despite being hard to scale up, offers flexibility in handling attribute editing cases. Therefore, we use generative model-based data to further fine-tune DreamVE. Besides, we design an efficient and powerful editing framework for DreamVE. We build on the SOTA T2V model and use a token concatenation with early drop approach to inject source image guidance, ensuring strong consistency and editability. The codes and models will be released.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
