arXiv 2312.067392023-12-11cs.CV

SmartEdit：探索基于多模态大语言模型的复杂指令图像编辑

SmartEdit: Exploring Complex Instruction-based Image Editing with Multimodal Large Language Models

University of Macau（澳门大学）；ARC Lab, Tencent PCG（腾讯平台与内容事业群ARC实验室）；Tencent AI Lab（腾讯人工智能实验室）；Shenzhen Institute of Advanced Technology（深圳先进技术研究院）；Shanghai Artificial Intelligence Laboratory（上海人工智能实验室）；The Chinese University of Hong Kong, Shenzhen（香港中文大学（深圳））；Shenzhen Research Institute of Big Data（深圳大数据研究院）；Tsinghua University（清华大学）

Yuzhou Huang, Liangbin Xie, Xintao Wang, Ziyang Yuan, Xiaodong Cun, Yixiao Ge, Jiantao Zhou, Chao Dong, Rui Huang, Ruimao Zhang, Ying Shan

备注：: Project page: https://yuzhou914.github.io/SmartEdit/

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

针对现有基于指令的图像编辑方法在复杂场景下效果不佳的问题，提出融合多模态大语言模型与双向交互模块的SmartEdit方法，并构建Reason-Edit数据集，实验证明其性能优于现有方法。

摘要：

当前基于指令的编辑方法（如InstructPix2Pix）在复杂场景下往往难以产生令人满意的结果，这是因为它们依赖扩散模型中简单的CLIP文本编码器。为解决这一问题，本文提出了SmartEdit——一种新型的基于指令的图像编辑方法，它利用多模态大语言模型（MLLMs）来增强理解与推理能力。然而，在需要复杂推理的场景中，直接整合这些要素仍面临挑战。为缓解这一问题，我们提出了双向交互模块，可实现输入图像与MLLM输出之间全面的双向信息交互。在训练过程中，我们首先引入感知数据，以提升扩散模型的感知与理解能力。随后我们证明，少量复杂指令编辑数据就能有效激发SmartEdit针对更复杂指令的编辑能力。我们进一步构建了一个全新的评估数据集Reason-Edit，专门适配复杂指令图像编辑任务。在该评估数据集上的定量与定性结果均表明，我们的SmartEdit性能优于以往方法，为复杂指令图像编辑的实际应用铺平了道路。

英文摘要：

Current instruction-based editing methods, such as InstructPix2Pix, often fail to produce satisfactory results in complex scenarios due to their dependence on the simple CLIP text encoder in diffusion models. To rectify this, this paper introduces SmartEdit, a novel approach to instruction-based image editing that leverages Multimodal Large Language Models (MLLMs) to enhance their understanding and reasoning capabilities. However, direct integration of these elements still faces challenges in situations requiring complex reasoning. To mitigate this, we propose a Bidirectional Interaction Module that enables comprehensive bidirectional information interactions between the input image and the MLLM output. During training, we initially incorporate perception data to boost the perception and understanding capabilities of diffusion models. Subsequently, we demonstrate that a small amount of complex instruction editing data can effectively stimulate SmartEdit's editing capabilities for more complex instructions. We further construct a new evaluation dataset, Reason-Edit, specifically tailored for complex instruction-based image editing. Both quantitative and qualitative results on this evaluation dataset indicate that our SmartEdit surpasses previous methods, paving the way for the practical application of complex instruction-based image editing.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。