arXiv 2503.198392025-03-29cs.CV

FireEdit：基于区域感知视觉语言模型的细粒度指令图像编辑

FireEdit: Fine-grained Instruction-based Image Editing via Region-aware Vision Language Model

Shenzhen Campus of Sun Yat-sen University（中山大学深圳校区）；Hunyuan, Tencent（腾讯混元）；Tsinghua University（清华大学）；HKUST（香港科技大学）

Jun Zhou, Jiahao Li, Zunnan Xu, Hanhui Li, Yiji Cheng, Fa-Ting Hong, Qin Lin, Qinglin Lu, Xiaodan Liang

备注：: Accepted to CVPR 2025

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

FireEdit提出基于区域感知视觉语言模型的细粒度指令图像编辑框架，通过区域标记、时间感知目标注入和混合视觉交叉注意力模块，提升复杂场景下的编辑精度与语义一致性，超越现有方法。

摘要：

目前，基于指令的图像编辑方法通过利用视觉语言模型（VLM）强大的跨模态理解能力，已取得显著进展。然而，它们仍面临三个关键领域的挑战：1）复杂场景；2）语义一致性；以及3）细粒度编辑。为解决这些问题，我们提出了FireEdit，一个创新的基于细粒度指令的图像编辑框架，利用区域感知VLM。FireEdit旨在准确理解用户指令，并确保对编辑过程的有效控制。具体而言，我们通过引入额外的区域标记来增强VLM的细粒度视觉感知能力。仅依赖LLM的输出引导扩散模型可能导致次优的编辑结果。因此，我们提出了时间感知目标注入模块和混合视觉交叉注意力模块。前者通过将时间步嵌入与文本嵌入集成，动态调整不同去噪阶段的引导强度。后者增强图像编辑的视觉细节，从而保持编辑结果与源图像之间的语义一致性。通过将增强细粒度区域标记的VLM与时间依赖的扩散模型相结合，FireEdit在理解编辑指令和保持高语义一致性方面展现出显著优势。大量实验表明，我们的方法超越了最先进的基于指令的图像编辑方法。我们的项目可在https://zjgans.github.io/fireedit.github.io获取。

英文摘要：

Currently, instruction-based image editing methods have made significant progress by leveraging the powerful cross-modal understanding capabilities of vision language models (VLMs). However, they still face challenges in three key areas: 1) complex scenarios; 2) semantic consistency; and 3) fine-grained editing. To address these issues, we propose FireEdit, an innovative Fine-grained Instruction-based image editing framework that exploits a REgion-aware VLM. FireEdit is designed to accurately comprehend user instructions and ensure effective control over the editing process. Specifically, we enhance the fine-grained visual perception capabilities of the VLM by introducing additional region tokens. Relying solely on the output of the LLM to guide the diffusion model may lead to suboptimal editing results. Therefore, we propose a Time-Aware Target Injection module and a Hybrid Visual Cross Attention module. The former dynamically adjusts the guidance strength at various denoising stages by integrating timestep embeddings with the text embeddings. The latter enhances visual details for image editing, thereby preserving semantic consistency between the edited result and the source image. By combining the VLM enhanced with fine-grained region tokens and the time-dependent diffusion model, FireEdit demonstrates significant advantages in comprehending editing instructions and maintaining high semantic consistency. Extensive experiments indicate that our approach surpasses the state-of-the-art instruction-based image editing methods. Our project is available at https://zjgans.github.io/fireedit.github.io.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。