FreeEdit：基于多模态指令的无掩码参考图像编辑
FreeEdit: Mask-free Reference-based Image Editing with Multi-modal Instruction
AI总结：
提出FreeEdit，一种基于多模态指令的无掩码参考图像编辑方法，通过DRRA模块增强细节重建，并构建FreeBench数据集实现高质量零样本编辑。
摘要：
在图像编辑中引入用户指定的视觉概念非常实用，因为这些概念比基于文本的描述能更准确地传达用户的意图。我们提出了FreeEdit，一种实现此类基于参考图像编辑的新方法，它可以根据用户友好的语言指令准确地从参考图像中重现视觉概念。我们的方法利用多模态指令编码器对语言指令进行编码以指导编辑过程。这种隐式定位编辑区域的方式消除了对手动编辑掩码的需求。为了增强参考细节的重建，我们引入了Decoupled Residual ReferAttention (DRRA)模块。该模块旨在以残差方式将细节提取器提取的细粒度参考特征集成到图像编辑过程中，而不干扰原始的自注意力机制。鉴于现有数据集不适用于基于参考的图像编辑任务，特别是由于构建包含参考图像的图像三元组的困难，我们使用新开发的双重重绘方案策划了一个高质量数据集FreeBench。FreeBench包含编辑前后的图像、详细的编辑指令以及保持被编辑对象身份的参考图像，涵盖对象添加、替换和删除等任务。通过在FreeBench上进行分阶段训练随后进行质量调优，FreeEdit通过便捷的语言指令实现了高质量的零样本编辑。我们进行了广泛实验以评估FreeEdit在多种任务类型上的有效性，证明其优于现有方法。代码将发布于：https://freeedit.github.io/。
英文摘要：
Introducing user-specified visual concepts in image editing is highly practical as these concepts convey the user's intent more precisely than text-based descriptions. We propose FreeEdit, a novel approach for achieving such reference-based image editing, which can accurately reproduce the visual concept from the reference image based on user-friendly language instructions. Our approach leverages the multi-modal instruction encoder to encode language instructions to guide the editing process. This implicit way of locating the editing area eliminates the need for manual editing masks. To enhance the reconstruction of reference details, we introduce the Decoupled Residual ReferAttention (DRRA) module. This module is designed to integrate fine-grained reference features extracted by a detail extractor into the image editing process in a residual way without interfering with the original self-attention. Given that existing datasets are unsuitable for reference-based image editing tasks, particularly due to the difficulty in constructing image triplets that include a reference image, we curate a high-quality dataset, FreeBench, using a newly developed twice-repainting scheme. FreeBench comprises the images before and after editing, detailed editing instructions, as well as a reference image that maintains the identity of the edited object, encompassing tasks such as object addition, replacement, and deletion. By conducting phased training on FreeBench followed by quality tuning, FreeEdit achieves high-quality zero-shot editing through convenient language instructions. We conduct extensive experiments to evaluate the effectiveness of FreeEdit across multiple task types, demonstrating its superiority over existing methods. The code will be available at: https://freeedit.github.io/.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
