MM-Diff：基于多模态条件集成的高保真图像个性化
MM-Diff: High-Fidelity Image Personalization via Multi-Modal Condition Integration
AI总结：
提出无调优图像个性化框架MM-Diff，通过视觉编码器提取CLS和patch嵌入并结合多模态交叉注意力机制，辅以交叉注意力图约束，实现单/多主体高保真图像的快速生成。
摘要：
近期基于扩散模型的无调优个性化图像生成取得了令人瞩目的进展。然而，为了提升主体保真度，现有方法要么重新训练扩散模型，要么向其中注入密集的视觉嵌入，两者都存在泛化性和效率差的问题。此外，由于不受约束的交叉注意力机制，这些方法在多主体图像生成方面表现不佳。在本文中，我们提出了MM-Diff，这是一个统一且无调优的图像个性化框架，能够在几秒钟内生成单主体和多主体的高保真图像。具体而言，为了同时增强文本一致性和主体保真度，MM-Diff采用视觉编码器将输入图像转换为CLS和patch嵌入。CLS嵌入一方面用于增强文本嵌入，另一方面与patch嵌入一起导出少量细节丰富的主体嵌入，两者均通过精心设计的多模态交叉注意力机制高效集成到扩散模型中。此外，MM-Diff在训练阶段引入了交叉注意力图约束，确保在推理过程中无需任何预定义输入（例如布局）即可进行灵活的多主体图像采样。大量实验表明，MM-Diff的性能优于其他领先方法。
英文摘要：
Recent advances in tuning-free personalized image generation based on diffusion models are impressive. However, to improve subject fidelity, existing methods either retrain the diffusion model or infuse it with dense visual embeddings, both of which suffer from poor generalization and efficiency. Also, these methods falter in multi-subject image generation due to the unconstrained cross-attention mechanism. In this paper, we propose MM-Diff, a unified and tuning-free image personalization framework capable of generating high-fidelity images of both single and multiple subjects in seconds. Specifically, to simultaneously enhance text consistency and subject fidelity, MM-Diff employs a vision encoder to transform the input image into CLS and patch embeddings. CLS embeddings are used on the one hand to augment the text embeddings, and on the other hand together with patch embeddings to derive a small number of detail-rich subject embeddings, both of which are efficiently integrated into the diffusion model through the well-designed multimodal cross-attention mechanism. Additionally, MM-Diff introduces cross-attention map constraints during the training phase, ensuring flexible multi-subject image sampling during inference without any predefined inputs (e.g., layout). Extensive experiments demonstrate the superior performance of MM-Diff over other leading methods.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
