Mod-Adapter：通过调制适配器实现无需微调的多概念个性化
Mod-Adapter: Tuning-Free and Versatile Multi-concept Personalization via Modulation Adapter
AI总结：
本文提出Mod-Adapter方法，通过调制适配器实现无需微调的多概念个性化，有效定制物体和抽象概念，提升生成质量。
摘要：
个性化文本到图像生成旨在合成用户提供的概念在多样上下文中的图像。尽管多概念个性化近年来取得了进展，但大多数方法局限于物体概念，并且在定制抽象概念（例如姿态、照明）时存在困难。一些方法开始探索支持抽象概念的多概念个性化，但它们需要为每个新概念进行测试时微调，这既耗时又容易在有限的训练图像上过拟合。在本文中，我们提出了一种新颖的无需微调的多概念个性化方法，能够在不进行测试时微调的情况下有效定制物体和抽象概念。我们的方法基于预训练扩散变压器（DiTs）模型中的调制机制，利用调制空间的局部性和语义意义特性。具体来说，我们提出了一种新的模块Mod-Adapter，用于预测概念特定的调制方向，以指导概念相关文本标记的调制过程。它引入了视觉-语言交叉注意力机制来提取概念视觉特征，并引入了混合专家（MoE）层，这些层能够自适应地将概念特征映射到调制空间。此外，为了解决概念图像空间与调制空间之间大差距导致的训练困难，我们引入了一种VLM引导的预训练策略，利用视觉-语言模型强大的图像理解能力来提供语义监督信号。为了进行全面比较，我们扩展了一个标准基准，将其扩展以包含抽象概念。我们的方法在多概念个性化中实现了最先进的性能，这得到了定量、定性和人类评估的支持。
英文摘要：
Personalized text-to-image generation aims to synthesize images of user-provided concepts in diverse contexts. Despite recent progress in multi-concept personalization, most are limited to object concepts and struggle to customize abstract concepts (e.g., pose, lighting). Some methods have begun exploring multi-concept personalization supporting abstract concepts, but they require test-time fine-tuning for each new concept, which is time-consuming and prone to overfitting on limited training images. In this work, we propose a novel tuning-free method for multi-concept personalization that can effectively customize both object and abstract concepts without test-time fine-tuning. Our method builds upon the modulation mechanism in pre-trained Diffusion Transformers (DiTs) model, leveraging the localized and semantically meaningful properties of the modulation space. Specifically, we propose a novel module, Mod-Adapter, to predict concept-specific modulation direction for the modulation process of concept-related text tokens. It introduces vision-language cross-attention for extracting concept visual features, and Mixture-of-Experts (MoE) layers that adaptively map the concept features into the modulation space. Furthermore, to mitigate the training difficulty caused by the large gap between the concept image space and the modulation space, we introduce a VLM-guided pre-training strategy that leverages the strong image understanding capabilities of vision-language models to provide semantic supervision signals. For a comprehensive comparison, we extend a standard benchmark by incorporating abstract concepts. Our method achieves state-of-the-art performance in multi-concept personalization, supported by quantitative, qualitative, and human evaluations.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
