TIE：革新基于文本的图像编辑，实现复杂提示跟随与高保真编辑
TIE: Revolutionizing Text-based Image Editing for Complex-Prompt Following and High-Fidelity Editing
AI总结：
本文提出TIE框架，利用多模态大语言模型的思维链推理和定位能力辅助扩散模型，通过指令分解、区域定位和详细描述及微调LISA模型，实现复杂提示下的高保真图像编辑，性能超越现有最先进模型。
摘要：
随着图像生成领域的快速发展，传统的扩散模型以及与多模态大语言模型（LLMs）集成的扩散模型在解释复杂提示和保持编辑前后图像一致性方面仍面临局限性。为了应对这些挑战，我们提出了一种创新的图像编辑框架，该框架利用多模态大语言模型强大的思维链（CoT）推理和定位能力来辅助扩散模型生成更精细的图像。我们首先精心设计了一个包含指令分解、区域定位和详细描述的思维链过程。随后，我们使用多模态大语言模型的思维链过程和编辑图像的掩码对LISA模型（一种轻量级多模态大语言模型）进行微调。通过向扩散模型提供生成的提示和图像掩码的知识，我们的模型生成的图像具有更优的指令理解能力。通过大量实验，我们的模型在图像生成方面表现出卓越的性能，超越了现有的最先进模型。值得注意的是，我们的模型展现出增强的理解复杂提示并生成相应图像的能力，同时在生成前后保持图像的高保真度和一致性。
英文摘要：
As the field of image generation rapidly advances, traditional diffusion models and those integrated with multimodal large language models (LLMs) still encounter limitations in interpreting complex prompts and preserving image consistency pre and post-editing. To tackle these challenges, we present an innovative image editing framework that employs the robust Chain-of-Thought (CoT) reasoning and localizing capabilities of multimodal LLMs to aid diffusion models in generating more refined images. We first meticulously design a CoT process comprising instruction decomposition, region localization, and detailed description. Subsequently, we fine-tune the LISA model, a lightweight multimodal LLM, using the CoT process of Multimodal LLMs and the mask of the edited image. By providing the diffusion models with knowledge of the generated prompt and image mask, our models generate images with a superior understanding of instructions. Through extensive experiments, our model has demonstrated superior performance in image generation, surpassing existing state-of-the-art models. Notably, our model exhibits an enhanced ability to understand complex prompts and generate corresponding images, while maintaining high fidelity and consistency in images before and after generation.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
