引导基于指令的图像编辑：多模态大语言模型
Guiding Instruction-based Image Editing via Multimodal Large Language Models
AI总结：
本文提出MGIE，利用多模态大语言模型生成表达性指令，引导图像编辑模型进行端到端训练，在自动指标和人工评估上显著提升性能。
摘要：
基于指令的图像编辑通过自然命令提高了图像操作的可控性和灵活性，无需详细描述或区域掩码。然而，人类指令有时过于简短，当前方法难以捕捉并遵循。多模态大语言模型（MLLMs）在跨模态理解和基于语言的视觉感知响应生成方面展现出有前景的能力。我们研究了MLLMs如何促进编辑指令，并提出了MLLM引导的图像编辑（MGIE）。MGIE学习推导出表达性指令并提供明确指导。编辑模型通过端到端训练联合捕捉这种视觉想象并执行操作。我们评估了Photoshop风格修改、全局照片优化和局部编辑等多个方面。大量实验结果表明，表达性指令对基于指令的图像编辑至关重要，我们的MGIE在自动指标和人工评估中能带来显著改进，同时保持有竞争力的推理效率。
英文摘要：
Instruction-based image editing improves the controllability and flexibility of image manipulation via natural commands without elaborate descriptions or regional masks. However, human instructions are sometimes too brief for current methods to capture and follow. Multimodal large language models (MLLMs) show promising capabilities in cross-modal understanding and visual-aware response generation via LMs. We investigate how MLLMs facilitate edit instructions and present MLLM-Guided Image Editing (MGIE). MGIE learns to derive expressive instructions and provides explicit guidance. The editing model jointly captures this visual imagination and performs manipulation through end-to-end training. We evaluate various aspects of Photoshop-style modification, global photo optimization, and local editing. Extensive experimental results demonstrate that expressive instructions are crucial to instruction-based image editing, and our MGIE can lead to a notable improvement in automatic metrics and human evaluation while maintaining competitive inference efficiency.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
