arXiv 2306.144352024-04-07cs.CVcs.LG

DragDiffusion：利用扩散模型进行基于点的交互式图像编辑

DragDiffusion: Harnessing Diffusion Models for Interactive Point-based Image Editing

National University of Singapore（新加坡国立大学）；ByteDance Inc.（字节跳动公司）

Yujun Shi, Chuhui Xue, Jun Hao Liew, Jiachun Pan, Hanshu Yan, Wenqing Zhang, Vincent Y. F. Tan, Song Bai

备注：: Code is released at https://github.com/Yujun-Shi/DragDiffusion

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

本文提出 DragDiffusion，将基于点的交互式编辑扩展到大规模预训练扩散模型，通过优化扩散潜变量并结合 LoRA 微调与 latent-MasaCtrl 保持原图身份，同时发布 DragBench 基准验证其通用性。

摘要：

准确且可控的图像编辑是一项具有挑战性的任务，近来受到广泛关注。值得注意的是，DragGAN 是一种基于点的交互式图像编辑框架，能够以像素级精度取得令人印象深刻的编辑结果。然而，由于其依赖生成对抗网络（GAN），其通用性受到预训练 GAN 模型能力的限制。在本工作中，我们将该编辑框架扩展到扩散模型，并提出一种名为 DragDiffusion 的新颖方法。通过利用大规模预训练扩散模型，我们显著提升了基于点的交互式编辑在真实图像和扩散生成图像上的适用性。我们的方法通过优化扩散潜变量来实现精确的空间控制。该优化过程的监督信号来自扩散模型的 UNet 特征，这些特征已知包含丰富的语义和几何信息。此外，我们引入了 LoRA 微调和 latent-MasaCtrl 两项附加技术，以进一步保持原始图像的身份特征。最后，我们提出了一个具有挑战性的基准数据集 DragBench——这是首个用于评估基于点的交互式图像编辑方法性能的基准。在多种挑战性场景（如包含多个对象、多样对象类别、各种风格等的图像）上的实验证明了 DragDiffusion 的多功能性和通用性。代码：https://github.com/Yujun-Shi/DragDiffusion。

英文摘要：

Accurate and controllable image editing is a challenging task that has attracted significant attention recently. Notably, DragGAN is an interactive point-based image editing framework that achieves impressive editing results with pixel-level precision. However, due to its reliance on generative adversarial networks (GANs), its generality is limited by the capacity of pretrained GAN models. In this work, we extend this editing framework to diffusion models and propose a novel approach DragDiffusion. By harnessing large-scale pretrained diffusion models, we greatly enhance the applicability of interactive point-based editing on both real and diffusion-generated images. Our approach involves optimizing the diffusion latents to achieve precise spatial control. The supervision signal of this optimization process is from the diffusion model's UNet features, which are known to contain rich semantic and geometric information. Moreover, we introduce two additional techniques, namely LoRA fine-tuning and latent-MasaCtrl, to further preserve the identity of the original image. Lastly, we present a challenging benchmark dataset called DragBench -- the first benchmark to evaluate the performance of interactive point-based image editing methods. Experiments across a wide range of challenging cases (e.g., images with multiple objects, diverse object categories, various styles, etc.) demonstrate the versatility and generality of DragDiffusion. Code: https://github.com/Yujun-Shi/DragDiffusion.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。