arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2409.00522cs.CV
收藏

EraseDraw:通过从图像中擦除对象来学习逐步绘制

EraseDraw: Learning to Draw Step-by-Step via Erasing Objects from Images

Columbia University(哥伦比亚大学)

Alper Canberk, Maksym Bondarenko, Ege Ozguroglu, Ruoshi Liu, Carl Vondrick

AI总结:

本文提出EraseDraw框架,通过反转对象移除过程生成高质量数据,训练文本条件扩散模型实现空间、物理和光学一致的对象插入,并结合CLIP引导的束搜索自动化迭代插入,在自然图像上取得最先进效果。

摘要:

绘画等创作过程通常涉及逐一创建图像的不同组成部分。我们能否构建一个计算模型来执行此任务?先前的工作常因对图像进行全局修改、将对象插入到不切实际的空间位置以及生成不准确的光照细节而失败。我们观察到,尽管最先进的模型在对象插入方面表现不佳,但它们能够很好地移除自然图像中的对象并擦除背景。通过反转对象移除的方向,我们获得了高质量的数据,用于学习插入在空间、物理和光学上与周围环境一致的对象。借助这种可扩展的自动数据生成管道,我们可以创建用于学习对象插入的数据集,并以此训练我们提出的文本条件扩散模型。定性和定量实验表明,我们的模型在对象插入方面取得了最先进的结果,尤其是对于自然图像。我们在跨不同领域的多样化插入提示和图像上展示了令人信服的结果。此外,我们通过将插入模型与由CLIP引导的束搜索相结合,实现了迭代插入的自动化。

英文摘要:

Creative processes such as painting often involve creating different components of an image one by one. Can we build a computational model to perform this task? Prior works often fail by making global changes to the image, inserting objects in unrealistic spatial locations, and generating inaccurate lighting details. We observe that while state-of-the-art models perform poorly on object insertion, they can remove objects and erase the background in natural images very well. Inverting the direction of object removal, we obtain high-quality data for learning to insert objects that are spatially, physically, and optically consistent with the surroundings. With this scalable automatic data generation pipeline, we can create a dataset for learning object insertion, which is used to train our proposed text conditioned diffusion model. Qualitative and quantitative experiments have shown that our model achieves state-of-the-art results in object insertion, particularly for in-the-wild images. We show compelling results on diverse insertion prompts and images across various domains.In addition, we automate iterative insertion by combining our insertion model with beam search guided by CLIP.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑