FlexEdit：将自由形状掩码与 VLLM 结合以实现灵活的图像编辑
FlexEdit: Marrying Free-Shape Masks to VLLM for Flexible Image Editing
AI总结：
FlexEdit 是一种端到端图像编辑方法，结合自由形状掩码与语言指令，利用 VLLM 和 Mask Enhance Adapter 实现灵活编辑，在 FSMI-Edit 基准上达到 SOTA 性能。
摘要：
将视觉大语言模型（VLLM）与扩散模型相结合，提供了一种基于人类语言指令执行图像编辑任务的强大方法。然而，仅靠语言指令往往无法准确传达用户需求，特别是当用户想要在图像特定区域添加或替换元素时。幸运的是，掩码能有效指示需要编辑的确切位置或元素，但要求用户在期望位置精确绘制形状，这对用户极不友好。为解决此问题，我们提出 FlexEdit，一种端到端的图像编辑方法，利用自由形状掩码和语言指令实现灵活编辑。我们的方法采用 VLLM 来理解图像内容、掩码和用户指令。此外，我们引入了 Mask Enhance Adapter（MEA），将 VLLM 的嵌入与图像数据融合，确保掩码信息与模型输出嵌入的无缝集成。此外，我们构建了 FSMI-Edit，一个专为自由形状掩码定制的基准，包含 8 种类型的自由形状掩码。大量实验表明，我们的方法在基于 LLM 的图像编辑中达到了最先进的（SOTA）性能，且我们简单的提示技术在有效性方面表现突出。代码和数据可在 https://github.com/A-new-b/flex_edit 找到。
英文摘要：
Combining Vision Large Language Models (VLLMs) with diffusion models offers a powerful method for executing image editing tasks based on human language instructions. However, language instructions alone often fall short in accurately conveying user requirements, particularly when users want to add, replace elements in specific areas of an image. Luckily, masks can effectively indicate the exact locations or elements to be edited, while they require users to precisely draw the shapes at the desired locations, which is highly user-unfriendly. To address this, we propose FlexEdit, an end-to-end image editing method that leverages both free-shape masks and language instructions for Flexible Editing. Our approach employs a VLLM in comprehending the image content, mask, and user instructions. Additionally, we introduce the Mask Enhance Adapter (MEA) that fuses the embeddings of the VLLM with the image data, ensuring a seamless integration of mask information and model output embeddings. Furthermore, we construct FSMI-Edit, a benchmark specifically tailored for free-shape mask, including 8 types of free-shape mask. Extensive experiments show that our method achieves state-of-the-art (SOTA) performance in LLM-based image editing, and our simple prompting technique stands out in its effectiveness. The code and data can be found at https://github.com/A-new-b/flex_edit.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
