IMG：通过隐式多模态引导校准扩散模型
IMG: Calibrating Diffusion Models via Implicit Multimodal Guidance
AI总结：
本文提出 IMG，一种无需额外数据或编辑、基于 MLLM 诊断与隐式条件特征调控的再生成对齐框架，可提升 SDXL、SDXL-DPO 和 FLUX 的图文对齐并兼容既有微调方法。
摘要：
确保扩散生成图像与输入提示之间精确的多模态对齐，一直是一项长期挑战。早期工作使用高质量偏好数据微调扩散权重，这类数据往往有限且难以扩展。近期基于编辑的方法会进一步细化生成图像的局部区域，但可能损害整体图像质量。在本文中，我们提出 Implicit Multimodal Guidance（IMG，隐式多模态引导），这是一种新颖的基于再生成的多模态对齐框架，无需额外数据或编辑操作。具体而言，给定一张生成图像及其提示，IMG：a）利用多模态大语言模型（MLLM）识别未对齐之处；b）引入一个 Implicit Aligner（隐式对齐器），通过操控扩散条件特征来减少未对齐并实现再生成；c）将重新对齐目标表述为一个可训练目标，即 Iteratively Updated Preference Objective（迭代更新偏好目标）。在 SDXL、SDXL-DPO 和 FLUX 上进行的大量定性与定量评估表明，IMG 优于现有对齐方法。此外，IMG 可作为灵活的即插即用适配器，无缝增强以往基于微调的对齐方法。我们的代码将在 https://github.com/SHI-Labs/IMG-Multimodal-Diffusion-Alignment 提供。
英文摘要：
Ensuring precise multimodal alignment between diffusion-generated images and input prompts has been a long-standing challenge. Earlier works finetune diffusion weight using high-quality preference data, which tends to be limited and difficult to scale up. Recent editing-based methods further refine local regions of generated images but may compromise overall image quality. In this work, we propose Implicit Multimodal Guidance (IMG), a novel re-generation-based multimodal alignment framework that requires no extra data or editing operations. Specifically, given a generated image and its prompt, IMG a) utilizes a multimodal large language model (MLLM) to identify misalignments; b) introduces an Implicit Aligner that manipulates diffusion conditioning features to reduce misalignments and enable re-generation; and c) formulates the re-alignment goal into a trainable objective, namely Iteratively Updated Preference Objective. Extensive qualitative and quantitative evaluations on SDXL, SDXL-DPO, and FLUX show that IMG outperforms existing alignment methods. Furthermore, IMG acts as a flexible plug-and-play adapter, seamlessly enhancing prior finetuning-based alignment methods. Our code will be available at https://github.com/SHI-Labs/IMG-Multimodal-Diffusion-Alignment.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
