arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2307.06925cs.CVcs.GRcs.LG
收藏

用于文本到图像模型快速个性化的领域无关调优编码器

Domain-Agnostic Tuning-Encoder for Fast Personalization of Text-To-Image Models

Tel Aviv University(特拉维夫大学);NVIDIA(英伟达);Reichman University (IDC)(赖赫曼大学(IDC))

Moab Arar, Rinon Gal, Yuval Atzmon, Gal Chechik, Daniel Cohen-Or, Ariel Shamir, Amit H. Bermano

AI总结:

本文提出领域无关的T2I个性化编码器,通过基于对比的正则化将预测token推向最近CLIP token,在无需专门数据集或先验信息的情况下实现高保真、可编辑且语义更强的SOTA个性化表示。

摘要:

文本到图像(T2I)个性化允许用户通过在自然语言提示中结合自己的视觉概念来引导创意图像生成过程。最近,基于编码器的技术已成为T2I个性化的一种有效新方法,减少了对多张图像和漫长训练时间的需求。然而,大多数现有编码器局限于单一类别领域,这阻碍了它们处理多样化概念的能力。在本工作中,我们提出了一种领域无关方法,该方法不需要任何专门数据集或关于个性化概念的先验信息。我们引入了一种新颖的基于对比的正则化技术,通过将预测的token推向其最近的现有CLIP token,在保持对目标概念特征的高保真度的同时,使预测的嵌入接近潜在空间的可编辑区域。我们的实验结果证明了该方法的有效性,并展示了学习到的token如何比未经正则化的模型预测的token更具语义性。这带来了更好的表示,实现了最先进的性能,同时比以往方法更灵活。

英文摘要:

Text-to-image (T2I) personalization allows users to guide the creative image generation process by combining their own visual concepts in natural language prompts. Recently, encoder-based techniques have emerged as a new effective approach for T2I personalization, reducing the need for multiple images and long training times. However, most existing encoders are limited to a single-class domain, which hinders their ability to handle diverse concepts. In this work, we propose a domain-agnostic method that does not require any specialized dataset or prior information about the personalized concepts. We introduce a novel contrastive-based regularization technique to maintain high fidelity to the target concept characteristics while keeping the predicted embeddings close to editable regions of the latent space, by pushing the predicted tokens toward their nearest existing CLIP tokens. Our experimental results demonstrate the effectiveness of our approach and show how the learned tokens are more semantic than tokens predicted by unregularized models. This leads to a better representation that achieves state-of-the-art performance while being more flexible than previous methods.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑