PAIR-Diffusion：一个全面的多模态对象级图像编辑器
PAIR-Diffusion: A Comprehensive Multimodal Object-Level Image Editor
AI总结：
本文提出 PAIR Diffusion 框架，将图像视为对象组合，通过控制各对象的结构与外观实现无需反演的多模态对象级编辑。
摘要：
生成式图像编辑最近见证了极快的发展。一些工作使用文本等高级条件，而其他工作则使用低级条件。然而，它们大多缺乏对图像中不同对象属性的细粒度控制，即对象级图像编辑。在本工作中，我们通过将图像视为各种对象的融合来处理该任务，并旨在以细粒度方式控制每个对象的属性。在这些属性中，我们认为结构和外观最直观易懂，也最适合用于编辑。我们提出了 PAIR Diffusion，这是一个通用框架，能够使扩散模型控制图像中每个对象的结构和外观属性。我们表明，对图像中每个对象的属性拥有控制权会带来全面的编辑能力。我们的框架允许对真实图像进行多种对象级编辑操作，例如基于参考图像的外观编辑、自由形状编辑、添加对象以及生成变体。得益于我们的设计，我们不需要任何反演步骤。此外，我们提出了多模态无分类器引导，当将我们的方法与基础扩散模型结合使用时，它能够同时使用参考图像和文本编辑图像。我们通过在无条件扩散模型和基础扩散模型上广泛评估我们的框架，验证了上述主张。代码和模型发布请参见 https://vidit98.github.io/publication/conference-paper/pair_diff.html。
英文摘要：
Generative image editing has recently witnessed extremely fast-paced growth. Some works use high-level conditioning such as text, while others use low-level conditioning. Nevertheless, most of them lack fine-grained control over the properties of the different objects present in the image, i.e. object-level image editing. In this work, we tackle the task by perceiving the images as an amalgamation of various objects and aim to control the properties of each object in a fine-grained manner. Out of these properties, we identify structure and appearance as the most intuitive to understand and useful for editing purposes. We propose PAIR Diffusion, a generic framework that can enable a diffusion model to control the structure and appearance properties of each object in the image. We show that having control over the properties of each object in an image leads to comprehensive editing capabilities. Our framework allows for various object-level editing operations on real images such as reference image-based appearance editing, free-form shape editing, adding objects, and variations. Thanks to our design, we do not require any inversion step. Additionally, we propose multimodal classifier-free guidance which enables editing images using both reference images and text when using our approach with foundational diffusion models. We validate the above claims by extensively evaluating our framework on both unconditional and foundational diffusion models. Please refer to https://vidit98.github.io/publication/conference-paper/pair_diff.html for code and model release.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
