arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2302.12228cs.CVcs.GRcs.LG
收藏

基于编码器的文本到图像模型快速个性化领域调整

Encoder-based Domain Tuning for Fast Personalization of Text-to-Image Models

Tel Aviv University(特拉维夫大学);NVIDIA(英伟达)

Rinon Gal, Moab Arar, Yuval Atzmon, Amit H. Bermano, Gal Chechik, Daniel Cohen-Or

摘要:

文本到图像个性化旨在教导预先训练的扩散模型来推理新颖的用户提供的概念,将它们嵌入到由自然语言提示引导的新场景中。然而,当前的个性化方法与冗长的训练时间、高存储要求或身份丢失作斗争。为了克服这些限制,我们提出了一个基于编码器的域调整方法。我们的关键见解是,通过对来自给定领域的大量概念进行欠拟合,我们可以提高泛化能力,并创建一个更适合快速添加来自同一领域的新概念的模型。具体来说,我们采用两个组件:首先,编码器将来自给定域的目标概念的单个图像(例如特定的面部)作为输入,并学习将其映射到表示概念的单词嵌入。第二,文本到图像模型的一组正则化权重偏移,学习如何有效地摄取额外的概念。这些组件一起用于指导学习看不见的概念,使我们能够仅使用单个图像和5个训练步骤来个性化模型-将个性化从几十分钟加速到几秒钟,同时保持质量。

英文摘要:

Text-to-image personalization aims to teach a pre-trained diffusion model to reason about novel, user provided concepts, embedding them into new scenes guided by natural language prompts. However, current personalization approaches struggle with lengthy training times, high storage requirements or loss of identity. To overcome these limitations, we propose an encoder-based domain-tuning approach. Our key insight is that by underfitting on a large set of concepts from a given domain, we can improve generalization and create a model that is more amenable to quickly adding novel concepts from the same domain. Specifically, we employ two components: First, an encoder that takes as an input a single image of a target concept from a given domain, e.g. a specific face, and learns to map it into a word-embedding representing the concept. Second, a set of regularized weight-offsets for the text-to-image model that learn how to effectively ingest additional concepts. Together, these components are used to guide the learning of unseen concepts, allowing us to personalize a model using only a single image and as few as 5 training steps - accelerating personalization from dozens of minutes to seconds, while preserving quality.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑