以程序形式进行图像编辑的扩散模型
Image Editing As Programs with Diffusion Models
AI总结：
本文提出IEAP框架，将复杂图像编辑指令分解为由VLM智能体编程的原子操作序列，基于共享DiT主干和轻量适配器实现，在多种编辑基准上显著超越现有方法。
摘要：
尽管扩散模型在文本到图像生成方面取得了显著成功，但它们在指令驱动的图像编辑中仍面临重大挑战。我们的研究揭示了一个关键挑战：这些模型在处理涉及大幅布局变化的结构不一致编辑时尤其困难。为弥补这一不足，我们提出了Image Editing As Programs（IEAP），这是一个基于Diffusion Transformer（DiT）架构构建的统一图像编辑框架。其核心思想是以还原论的视角处理指令式编辑，将复杂的编辑指令分解为一系列原子操作。每个操作通过一个轻量级适配器实现，共享同一个DiT主干网络，并针对特定类型的编辑进行专门化设计。这些操作由基于视觉-语言模型（VLM）的智能体进行编程，协同支持任意且结构不一致的变换。通过以这种方式对编辑进行模块化和序列化，IEAP能够在广泛的编辑任务中稳健地泛化，从简单调整到大幅结构变化。大量实验表明，IEAP在各种编辑场景的标准基准上显著优于最先进的方法。在这些评估中，我们的框架提供了卓越的准确性和语义保真度，尤其是在复杂的多步指令方面。代码可在https://github.com/YujiaHu1109/IEAP获取。
英文摘要：
While diffusion models have achieved remarkable success in text-to-image generation, they encounter significant challenges with instruction-driven image editing. Our research highlights a key challenge: these models particularly struggle with structurally inconsistent edits that involve substantial layout changes. To mitigate this gap, we introduce Image Editing As Programs (IEAP), a unified image editing framework built upon the Diffusion Transformer (DiT) architecture. At its core, IEAP approaches instructional editing through a reductionist lens, decomposing complex editing instructions into sequences of atomic operations. Each operation is implemented via a lightweight adapter sharing the same DiT backbone and is specialized for a specific type of edit. Programmed by a vision-language model (VLM)-based agent, these operations collaboratively support arbitrary and structurally inconsistent transformations. By modularizing and sequencing edits in this way, IEAP generalizes robustly across a wide range of editing tasks, from simple adjustments to substantial structural changes. Extensive experiments demonstrate that IEAP significantly outperforms state-of-the-art methods on standard benchmarks across various editing scenarios. In these evaluations, our framework delivers superior accuracy and semantic fidelity, particularly for complex, multi-step instructions. Codes are available at https://github.com/YujiaHu1109/IEAP.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
