Creative processes such as painting often involve creating different components of an image one by one. Can we build a computational model to perform this task? Prior works often fail by making global changes to the image, inserting objects in unrealistic spatial locations, and generating inaccurate lighting details. We observe that while state-of-the-art models perform poorly on object insertion, they can remove objects and erase the background in natural images very well. Inverting the direction of object removal, we obtain high-quality data for learning to insert objects that are spatially, physically, and optically consistent with the surroundings. With this scalable automatic data generation pipeline, we can create a dataset for learning object insertion, which is used to train our proposed text conditioned diffusion model. Qualitative and quantitative experiments have shown that our model achieves state-of-the-art results in object insertion, particularly for in-the-wild images. We show compelling results on diverse insertion prompts and images across various domains.In addition, we automate iterative insertion by combining our insertion model with beam search guided by CLIP.
You Only Erase Once: Erasing Anything without Bringing Unexpected Content
School of Computer Science and Engineering, Sun Yat-sen University, China(中山大学计算机科学与工程学院);Amazon(亚马逊);Key Laboratory of Machine Intelligence and Advanced Computing, Ministry of Education, China(教育部机器智能与先进计算重点实验室)
Tell, Draw, and Repeat: Generating and Modifying Images Based on Continual Linguistic Instruction
University of Guelph(圭尔夫大学);Microsoft Research(微软研究院);Montreal Institute for Learning Algorithms(蒙特利尔学习算法研究所);Vector Institute for Artificial Intelligence(矢量人工智能研究所);University of Montreal(蒙特利尔大学);Canadian Institute for Advanced Research(加拿大高级研究院)