arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2409.08248cs.CV
收藏

TextBoost: 通过文本编码器提升文本到图像生成的个性化

TextBoost: Boosting Text Encoder for Personalized Text-to-Image Generation

KAIST(韩国科学技术院)

NaHyeon Park, Kunhee Kim, Hyunjung Shim

AI总结:

本文提出TextBoost,一种高效的文本到图像扩散模型单次个性化方法,通过仅微调文本编码器提升计算和存储效率,并保持语义完整性,从而实现更快收敛和更低存储需求,同时保持高质量生成。

摘要:

在本文中,我们介绍了TextBoost,一种高效的文本到图像扩散模型单次个性化方法。传统个性化方法通常涉及微调模型的大量部分,导致存储需求大且收敛慢。相反,我们提出仅选择性地微调文本编码器,显著提高了计算和存储效率。为了保持原始语义完整性,我们开发了一种新颖的因果保持适应机制。此外,轻量级适配器被用于在文本嵌入与交叉注意层交互之前局部细化文本嵌入,从而在极小的计算开销下显著增强文本嵌入的表达能力。在多样化的概念上进行的实证评估表明,TextBoost通过减少可训练参数的数量实现了更快的收敛速度和显著的存储需求降低。此外,TextBoost在主体保真度、文本保真度和生成多样性方面与现有方法相比具有可比性。我们展示所提出的方法为高质量文本到图像个性化提供了一种高效、可扩展且实用的解决方案,尤其在资源受限的环境中具有优势。

英文摘要:

In this paper, we introduce TextBoost, an efficient one-shot personalization approach for text-to-image diffusion models. Traditional personalization methods typically involve fine-tuning extensive portions of the model, leading to substantial storage requirements and slow convergence. In contrast, we propose selectively fine-tuning only the text encoder, significantly improving computational and storage efficiency. To preserve the original semantic integrity, we develop a novel causality-preserving adaptation mechanism. Additionally, lightweight adapters are employed to locally refine text embeddings immediately before their interaction with cross-attention layers, greatly enhancing the expressiveness of text embeddings with minimal computational overhead. Empirical evaluations across diverse concepts demonstrate that TextBoost achieves faster convergence and substantially reduces storage demands by minimizing the number of trainable parameters. Furthermore, TextBoost maintains comparable subject fidelity, superior text fidelity, and greater generation diversity compared to existing methods. We show that our proposed method offers an efficient, scalable, and practically applicable solution for high-quality text-to-image personalization, particularly beneficial in resource-constrained environments.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑