GeoDiffuser：基于几何的扩散模型图像编辑
GeoDiffuser: Geometry-Based Image Editing with Diffusion Models
AI总结：
GeoDiffuser提出将图像编辑视为几何变换并融入扩散模型注意力层，以零样本优化方式统一实现二维和三维对象平移、旋转与移除等编辑任务。
摘要：
图像生成模型的成功使我们能够构建基于文本或其他用户输入来编辑图像的方法。然而，这些方法往往是定制化的、不精确的、需要额外信息，或者仅限于二维图像编辑。我们提出了GeoDiffuser，这是一种基于优化的零样本方法，将常见的二维和三维基于图像的对象编辑能力统一到单一方法中。我们的关键见解是将图像编辑操作视为几何变换。我们证明这些变换可以直接融入扩散模型的注意力层中，以隐式执行编辑操作。我们的免训练优化方法使用一个目标函数，该函数在保持对象风格的同时生成合理的图像，例如具有准确的光照和阴影。它还会对对象原本所在位置被遮挡的部分进行图像修复。给定自然图像和用户输入，我们使用SAM分割前景对象，并估计相应的变换，该变换被我们的优化方法用于编辑。GeoDiffuser可以执行常见的二维和三维编辑，如对象平移、三维旋转和移除。我们展示了定量结果，包括一项感知研究，表明我们的方法优于现有方法。请访问https://ivl.cs.brown.edu/research/geodiffuser.html了解更多信息。
英文摘要：
The success of image generative models has enabled us to build methods that can edit images based on text or other user input. However, these methods are bespoke, imprecise, require additional information, or are limited to only 2D image edits. We present GeoDiffuser, a zero-shot optimization-based method that unifies common 2D and 3D image-based object editing capabilities into a single method. Our key insight is to view image editing operations as geometric transformations. We show that these transformations can be directly incorporated into the attention layers in diffusion models to implicitly perform editing operations. Our training-free optimization method uses an objective function that seeks to preserve object style but generate plausible images, for instance with accurate lighting and shadows. It also inpaints disoccluded parts of the image where the object was originally located. Given a natural image and user input, we segment the foreground object using SAM and estimate a corresponding transform which is used by our optimization approach for editing. GeoDiffuser can perform common 2D and 3D edits like object translation, 3D rotation, and removal. We present quantitative results, including a perceptual study, that shows how our approach is better than existing methods. Visit https://ivl.cs.brown.edu/research/geodiffuser.html for more information.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
