Infusion：防止定制化文本到图像扩散模型的过拟合
Infusion: Preventing Customized Text-to-Image Diffusion from Overfitting
AI总结：
本文分析了文本到图像定制中的概念过拟合问题，提出仅需11KB参数的Infusion方法，有效克服有限训练模态限制并保留非定制知识，在单/多概念生成上优于现有方法。
摘要：
文本到图像（T2I）定制旨在创建能够体现文本描述中所描绘特定视觉概念的图像。然而，现有研究仍面临一个主要挑战，即概念过拟合。为应对此挑战，我们首先分析了过拟合现象，将其分为概念无关过拟合（破坏非定制概念知识）和概念特定过拟合（局限于在有限模态上进行定制，即背景、布局、风格）。为评估过拟合程度，我们进一步引入了两个指标，即Latent Fisher散度和Wasserstein度量，分别用于衡量非定制和定制概念分布的变化。基于上述分析，我们提出了Infusion，这是一种T2I定制方法，能够学习目标概念以避免受限于有限的训练模态，同时保留非定制知识。值得注意的是，Infusion以极高的效率实现了这一目标，仅需11KB的训练参数。大量实验也表明，我们的方法在单概念和多概念定制生成中均优于现有最先进的方法。
英文摘要：
Text-to-image (T2I) customization aims to create images that embody specific visual concepts delineated in textual descriptions. However, existing works still face a main challenge, concept overfitting. To tackle this challenge, we first analyze overfitting, categorizing it into concept-agnostic overfitting, which undermines non-customized concept knowledge, and concept-specific overfitting, which is confined to customize on limited modalities, i.e, backgrounds, layouts, styles. To evaluate the overfitting degree, we further introduce two metrics, i.e, Latent Fisher divergence and Wasserstein metric to measure the distribution changes of non-customized and customized concept respectively. Drawing from the analysis, we propose Infusion, a T2I customization method that enables the learning of target concepts to avoid being constrained by limited training modalities, while preserving non-customized knowledge. Remarkably, Infusion achieves this feat with remarkable efficiency, requiring a mere 11KB of trained parameters. Extensive experiments also demonstrate that our approach outperforms state-of-the-art methods in both single and multi-concept customized generation.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
