MCIE: 多模态大语言模型驱动的复杂指令图像编辑与空间引导
MCIE: Multimodal LLM-Driven Complex Instruction Image Editing with Spatial Guidance
AI总结：
MCIE通过空间引导和背景一致性模块，提升复杂指令图像编辑的指令合规性，实现23.96%的性能提升。
摘要：
最近在基于指令的图像编辑领域取得了显著进展。然而，现有方法仍局限于相对简单的编辑操作，阻碍了需要复杂和组合性指令的现实应用。在本工作中，我们从架构设计、数据和评估协议的角度解决这些限制。具体来说，我们识别出当前模型的两个关键挑战：指令合规性不足和背景不一致。为此，我们提出了MCIE-E1，一种多模态大语言模型驱动的复杂指令图像编辑方法，集成了两个关键模块：空间感知的交叉注意力模块和背景一致的交叉注意力模块。前者通过在去噪过程中利用空间引导，显式地将语义指令与空间区域对齐，从而增强指令遵循能力；后者在未编辑区域保留特征以维持背景一致性。为了实现有效的训练，我们构建了一个专用的数据管道，以缓解复杂指令图像编辑数据集稀缺的问题，结合精细的自动过滤（通过强大的MLLM）和严格的真人验证。最后，为了全面评估基于复杂指令的图像编辑，我们引入了CIE-Bench，一个新的基准，包含两个新的评估指标。在CIE-Bench上的实验结果表明，MCIE-E1在定量和定性评估中均优于先前的最先进方法，实现了23.96%的指令合规性提升。
英文摘要：
Recent advances in instruction-based image editing have shown remarkable progress. However, existing methods remain limited to relatively simple editing operations, hindering real-world applications that require complex and compositional instructions. In this work, we address these limitations from the perspectives of architectural design, data, and evaluation protocols. Specifically, we identify two key challenges in current models: insufficient instruction compliance and background inconsistency. To this end, we propose MCIE-E1, a Multimodal Large Language Model-Driven Complex Instruction Image Editing method that integrates two key modules: a spatial-aware cross-attention module and a background-consistent cross-attention module. The former enhances instruction-following capability by explicitly aligning semantic instructions with spatial regions through spatial guidance during the denoising process, while the latter preserves features in unedited regions to maintain background consistency. To enable effective training, we construct a dedicated data pipeline to mitigate the scarcity of complex instruction-based image editing datasets, combining fine-grained automatic filtering via a powerful MLLM with rigorous human validation. Finally, to comprehensively evaluate complex instruction-based image editing, we introduce CIE-Bench, a new benchmark with two new evaluation metrics. Experimental results on CIE-Bench demonstrate that MCIE-E1 consistently outperforms previous state-of-the-art methods in both quantitative and qualitative assessments, achieving a 23.96% improvement in instruction compliance.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
