图像修复模型是指令引导图像编辑的有效工具
Image Inpainting Models are Effective Tools for Instruction-guided Image Editing
AI总结：
本文提出基于图像修复的指令引导图像编辑方法IIIE，通过四步流程结合语言模型与修复模型，无需联合微调即可实现高成功率与良好视觉质量。
摘要：
本文是CVPR2024 GenAI媒体生成挑战赛研讨会指令引导图像编辑赛道获胜方案的技术报告。指令引导的图像编辑近年来得到了广泛研究。最先进的方法，如SmartEdit和MGIE，通常通过联合训练将大语言模型与扩散模型相结合，前者提供文本理解能力，后者提供图像生成能力。然而，在我们的实验中，我们发现简单地将大语言模型和图像生成模型通过诸如掩码之类的中间引导连接起来，而不是进行联合微调，可以获得更好的编辑性能和成功率。我们使用一个四步流程IIIE（基于修复的指令引导图像编辑）：编辑类别分类、主要编辑对象识别、编辑掩码获取和图像修复。结果表明，通过语言模型和图像修复模型的适当组合，我们的流程可以达到高成功率和令人满意的视觉质量。
英文摘要：
This is the technique report for the winning solution of the CVPR2024 GenAI Media Generation Challenge Workshop's Instruction-guided Image Editing track. Instruction-guided image editing has been largely studied in recent years. The most advanced methods, such as SmartEdit and MGIE, usually combine large language models with diffusion models through joint training, where the former provides text understanding ability, and the latter provides image generation ability. However, in our experiments, we find that simply connecting large language models and image generation models through intermediary guidance such as masks instead of joint fine-tuning leads to a better editing performance and success rate. We use a 4-step process IIIE (Inpainting-based Instruction-guided Image Editing): editing category classification, main editing object identification, editing mask acquisition, and image inpainting. Results show that through proper combinations of language models and image inpainting models, our pipeline can reach a high success rate with satisfying visual quality.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
