通过大语言模型推理理解隐式用户意图以进行图像编辑
Understanding the Implicit User Intention via Reasoning with Large Language Model for Image Editing
AI总结：
提出CIELR方法，利用大语言模型推理将复杂图像编辑指令转化为简单动作，并通过迭代更新机制构建细粒度视觉表示，无需联合微调模型，在多个基准上显著超越现有方法。
摘要：
现有的图像编辑方法能够很好地处理简单的编辑指令。为了处理复杂的编辑指令，它们通常需要联合微调大语言模型（LLMs）和扩散模型（DMs），这涉及极高的计算复杂度和训练成本。为了解决这个问题，我们提出了一种新方法，称为基于LLM推理的复杂图像编辑（CIELR），该方法将复杂的用户指令转换为一组简单且明确的编辑动作，从而消除了联合微调大语言模型和扩散模型的需要。具体而言，我们首先使用基础模型构建输入图像的结构化语义表示。然后，我们引入一种迭代更新机制，能够逐步细化该表示，从而获得图像场景的细粒度视觉表示。这使我们能够执行复杂且灵活的图像编辑任务。在SmartEdit Reasoning Scenario Set上的大量实验表明，我们的方法在PSNR上超越了先前最先进的方法9.955 dB，这表明其在保持应一致区域方面具有卓越的保留能力。由于带有推理的复杂图像编辑公共数据集样本数量有限，我们构建了一个名为CIEBench的基准，包含86个图像样本，并配有一个专门用于基于推理的图像编辑的指标。CIELR在该基准上也优于先前的方法。代码和数据集可在https://github.com/Jia-shao/Reasoning-Editing获取。
英文摘要：
Existing image editing methods can handle simple editing instructions very well. To deal with complex editing instructions, they often need to jointly fine-tune the large language models (LLMs) and diffusion models (DMs), which involves very high computational complexity and training cost. To address this issue, we propose a new method, called \textbf{C}omplex \textbf{I}mage \textbf{E}diting via \textbf{L}LM \textbf{R}easoning (CIELR), which converts a complex user instruction into a set of simple and explicit editing actions, eliminating the need for jointly fine-tuning the large language models and diffusion models. Specifically, we first construct a structured semantic representation of the input image using foundation models. Then, we introduce an iterative update mechanism that can progressively refine this representation, obtaining a fine-grained visual representation of the image scene. This allows us to perform complex and flexible image editing tasks. Extensive experiments on the SmartEdit Reasoning Scenario Set show that our method surpasses the previous state-of-the-art by 9.955 dB in PSNR, indicating its superior preservation of regions that should remain consistent. Due to the limited number of samples of public datasets of complex image editing with reasoning, we construct a benchmark named CIEBench, containing 86 image samples, together with a metric specifically for reasoning-based image editing. CIELR also outperforms previous methods on this benchmark. The code and dataset are available at \href{https://github.com/Jia-shao/Reasoning-Editing}{https://github.com/Jia-shao/Reasoning-Editing}.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
