ELITE：将视觉概念编码到文本嵌入中以实现定制化文本到图像生成
ELITE: Encoding Visual Concepts into Textual Embeddings for Customized Text-to-Image Generation
AI总结：
针对定制化文生图中优化方法计算与内存开销大的问题，提出含全局与局部映射网络的学习型编码器 ELITE，实现更快的高保真反演与稳健可编辑生成。
摘要：
除了在想象创作方面具备前所未有的能力外，大型文本到图像模型还被期望能够在图像生成中接受定制化概念。现有工作通常以基于优化的方式学习此类概念，但这会带来过高的计算或内存负担。本文转而提出一种基于学习的编码器，该编码器由全局映射网络和局部映射网络组成，用于快速、准确的定制化文本到图像生成。具体而言，全局映射网络将给定图像的分层特征投影到文本词嵌入空间中的多个新词上，即一个用于良好可编辑概念的主词，以及用于排除无关干扰（例如背景）的其他辅助词。同时，局部映射网络将编码后的图像块特征注入交叉注意力层，以提供遗漏的细节，同时不牺牲主概念的可编辑性。我们在多种用户定义概念上将本文方法与现有基于优化的方法进行比较，证明本文方法能够实现高保真反演和更稳健的可编辑性，并且编码过程显著更快。我们的代码已公开发布于 https://github.com/csyxwei/ELITE。
英文摘要：
In addition to the unprecedented ability in imaginary creation, large text-to-image models are expected to take customized concepts in image generation. Existing works generally learn such concepts in an optimization-based manner, yet bringing excessive computation or memory burden. In this paper, we instead propose a learning-based encoder, which consists of a global and a local mapping networks for fast and accurate customized text-to-image generation. In specific, the global mapping network projects the hierarchical features of a given image into multiple new words in the textual word embedding space, i.e., one primary word for well-editable concept and other auxiliary words to exclude irrelevant disturbances (e.g., background). In the meantime, a local mapping network injects the encoded patch features into cross attention layers to provide omitted details, without sacrificing the editability of primary concepts. We compare our method with existing optimization-based approaches on a variety of user-defined concepts, and demonstrate that our method enables high-fidelity inversion and more robust editability with a significantly faster encoding process. Our code is publicly available at https://github.com/csyxwei/ELITE.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
