MIGE：相互增强的多模态基于指令的图像生成与编辑
MIGE: Mutually Enhanced Multimodal Instruction-Based Image Generation and Editing
AI总结：
本文提出统一框架 MIGE，通过多模态指令、共享输入输出形式和特征融合联合训练主体驱动生成与指令编辑，实现跨任务增强与泛化，并在新任务上达到 SOTA。
摘要：
尽管基于扩散的图像生成已取得显著进展，但主体驱动生成和基于指令的编辑仍具挑战性。现有方法通常将二者分开处理，受限于有限的高质量数据和较差的泛化能力。然而，两项任务都需要捕捉复杂的视觉变化，同时保持输入与输出之间的一致性。受此启发，我们提出 MIGE，这是一个利用多模态指令统一任务表示的框架。它首先将主体驱动生成视为在空白画布上进行创作，将基于指令的编辑视为对现有图像进行修改，建立共享的输入—输出表述；随后引入一种新型多模态编码器，将自由形式的多模态指令映射到统一的视觉—语言空间，并通过特征融合机制整合视觉与语义特征。这种统一使两项任务能够联合训练，并带来两个关键优势：（1）跨任务增强：通过利用共享的视觉和语义表示，联合训练提升了主体驱动生成与基于指令编辑中的指令遵循度和视觉一致性。（2）泛化能力：以统一格式学习促进了跨任务知识迁移，使 MIGE 能够泛化到新的组合任务，包括基于指令的主体驱动编辑。实验表明，MIGE 在主体驱动生成和基于指令编辑方面均表现优异，并在基于指令的主体驱动编辑这一新任务上达到 SOTA。代码和模型已公开发布于 https://github.com/Eureka-Maggie/MIGE。
英文摘要：
Despite significant progress in diffusion-based image generation, subject-driven generation and instruction-based editing remain challenging. Existing methods typically treat them separately, struggling with limited high-quality data and poor generalization. However, both tasks require capturing complex visual variations while maintaining consistency between inputs and outputs. Inspired by this, we propose MIGE, a unified framework that standardizes task representations using multimodal instructions. It first treats subject-driven generation as creation on a blank canvas and instruction-based editing as modification of an existing image, establishing a shared input-output formulation, then introduces a novel multimodal encoder that maps free-form multimodal instructions into a unified vision-language space, integrating visual and semantic features through a feature fusion mechanism. This unification enables joint training of both tasks, providing two key advantages: (1) Cross-Task Enhancement: by leveraging shared visual and semantic representations, joint training improves instruction adherence and visual consistency in both subject-driven generation and instruction-based editing. (2) Generalization: learning in a unified format facilitates cross-task knowledge transfer, enabling MIGE to generalize to novel compositional tasks, including instruction-based subject-driven editing. Experiments show that MIGE excels in both subject-driven generation and instruction-based editing while setting a SOTA in the new task of instruction-based subject-driven editing. Code and model have been publicly available at https://github.com/Eureka-Maggie/MIGE.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
