arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2305.18047cs.CV

InstructEdit：利用用户指令改进基于扩散模型的图像编辑自动掩码

InstructEdit: Improving Automatic Masks for Diffusion-based Image Editing With User Instructions

KAUST（阿卜杜拉国王科技大学）

Qian Wang, Biao Zhang, Michael Birsak, Peter Wonka

阅读 PDF
arXiv 页面 ↗HTML 原文 ↗

AI总结：

本文提出 InstructEdit 框架，通过语言处理器、分割器和图像编辑器协同工作，利用用户指令改进扩散模型图像编辑中的自动掩码质量，在复杂或多物体场景下实现更精确的细粒度编辑。

摘要：

近期研究探索了使用扩散模型进行文本引导的图像编辑，并根据文本提示生成编辑后的图像。然而，这些模型难以准确定位需要编辑的区域，也难以忠实地执行精确编辑。在本工作中，我们提出了一个名为 InstructEdit 的框架，能够基于用户指令进行细粒度编辑。我们提出的框架包含三个组件：语言处理器、分割器和图像编辑器。第一个组件语言处理器使用大型语言模型处理用户指令，其处理目标是解析用户指令，并为分割器输出提示、为图像编辑器输出描述文本。我们在该步骤中采用 ChatGPT，并可选地使用 BLIP2。第二个组件分割器使用语言处理器提供的分割提示，我们采用最先进的分割框架 Grounded Segment Anything，根据分割提示自动生成高质量掩码。第三个组件图像编辑器使用来自语言处理器的描述文本和来自分割器的掩码来计算编辑后的图像，我们采用 Stable Diffusion 和 DiffEdit 中的掩码引导生成来实现这一目的。实验表明，在输入图像包含复杂物体或多个物体的细粒度编辑应用中，我们的方法优于以往的编辑方法。我们相比 DiffEdit 提高了掩码质量，从而提高了编辑图像的质量。我们还表明，我们的框架可以接受多种形式的用户指令作为输入。代码见 https://github.com/QianWangX/InstructEdit。

英文摘要：

Recent works have explored text-guided image editing using diffusion models and generated edited images based on text prompts. However, the models struggle to accurately locate the regions to be edited and faithfully perform precise edits. In this work, we propose a framework termed InstructEdit that can do fine-grained editing based on user instructions. Our proposed framework has three components: language processor, segmenter, and image editor. The first component, the language processor, processes the user instruction using a large language model. The goal of this processing is to parse the user instruction and output prompts for the segmenter and captions for the image editor. We adopt ChatGPT and optionally BLIP2 for this step. The second component, the segmenter, uses the segmentation prompt provided by the language processor. We employ a state-of-the-art segmentation framework Grounded Segment Anything to automatically generate a high-quality mask based on the segmentation prompt. The third component, the image editor, uses the captions from the language processor and the masks from the segmenter to compute the edited image. We adopt Stable Diffusion and the mask-guided generation from DiffEdit for this purpose. Experiments show that our method outperforms previous editing methods in fine-grained editing applications where the input image contains a complex object or multiple objects. We improve the mask quality over DiffEdit and thus improve the quality of edited images. We also show that our framework can accept multiple forms of user instructions as input. We provide the code at https://github.com/QianWangX/InstructEdit.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。

↑