FramePainter：赋予交互式图像编辑视频扩散先验能力
FramePainter: Endowing Interactive Image Editing with Video Diffusion Priors
AI总结：
该研究将交互式图像编辑重构为图生视频生成任务，提出基于Stable Video Diffusion的FramePainter框架，通过轻量稀疏控制编码器和匹配注意力机制，以更少训练数据实现更优的编辑效果与跨场景泛化能力。
摘要：
交互式图像编辑允许用户通过绘制、点击、拖拽等视觉交互操作修改图像。现有方法从视频中构建此类监督信号，因为视频捕捉了物体在各种物理交互下的变化方式。然而，这些模型通常基于文生图扩散模型构建，因此需要（i）大量训练样本，以及（ii）额外的参考编码器，以学习真实世界动态和视觉一致性。\n在本文中，我们将该任务重新定义为图生视频生成问题，从而继承强大的视频扩散先验，以降低训练成本并确保时间一致性。具体而言，我们提出FramePainter作为该范式的高效实现方案。它以Stable Video Diffusion（稳定视频扩散模型）为初始化基础，仅使用一个轻量级稀疏控制编码器注入编辑信号。考虑到时间注意力在处理两帧间大幅运动时的局限性，我们进一步提出匹配注意力机制，在扩大感受野的同时增强编辑图像token与源图像token之间的密集对应关系。\n我们在多种编辑信号下验证了FramePainter的有效性和高效性：它以远少于现有方法的训练数据，显著优于此前的最优方法，实现了高度无缝连贯的图像编辑效果，例如自动调整杯子的倒影。此外，FramePainter在真实视频中未出现的场景中也展现出卓越的泛化能力，例如将小丑鱼变形为鲨鱼形状。我们的代码将发布于https://github.com/YBYBZhang/FramePainter。
英文摘要：
Interactive image editing allows users to modify images through visual interaction operations such as drawing, clicking, and dragging. Existing methods construct such supervision signals from videos, as they capture how objects change with various physical interactions. However, these models are usually built upon text-to-image diffusion models, so necessitate (i) massive training samples and (ii) an additional reference encoder to learn real-world dynamics and visual consistency. In this paper, we reformulate this task as an image-to-video generation problem, so that inherit powerful video diffusion priors to reduce training costs and ensure temporal consistency. Specifically, we introduce FramePainter as an efficient instantiation of this formulation. Initialized with Stable Video Diffusion, it only uses a lightweight sparse control encoder to inject editing signals. Considering the limitations of temporal attention in handling large motion between two frames, we further propose matching attention to enlarge the receptive field while encouraging dense correspondence between edited and source image tokens. We highlight the effectiveness and efficiency of FramePainter across various of editing signals: it domainantly outperforms previous state-of-the-art methods with far less training data, achieving highly seamless and coherent editing of images, \eg, automatically adjust the reflection of the cup. Moreover, FramePainter also exhibits exceptional generalization in scenarios not present in real-world videos, \eg, transform the clownfish into shark-like shape. Our code will be available at https://github.com/YBYBZhang/FramePainter.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
