StyleTokenizer：通过单个实例定义图像风格以控制扩散模型
StyleTokenizer: Defining Image Style by a Single Instance for Controlling Diffusion Models
AI总结：
本文提出StyleTokenizer方法，通过风格分词器将单张参考图像的风格表示与文本表示对齐，实现零样本风格控制图像生成，并构建Style30k数据集训练风格特征提取器，有效解决了图像风格控制与文本语义保持之间的冲突。
摘要：
尽管控制扩散过程的创新方法层出不穷，但在文本到图像生成中有效控制图像风格仍然是一项具有挑战性的任务。许多基于适配器的方法将图像表示条件施加到去噪过程上以实现图像控制。然而，这些条件与词嵌入空间不对齐，导致图像控制条件和文本控制条件之间产生干扰，并可能造成文本提示中语义信息的丢失。解决这一问题涉及两个关键挑战。首先，如何在控制中注入风格表示而不损害文本表示的有效性。其次，如何从单个参考图像中获得准确的风格表示。为了应对这些挑战，我们提出了StyleTokenizer，这是一种零样本风格控制图像生成方法，利用风格分词器将风格表示与文本表示对齐。这种对齐有效地最小化了对文本提示有效性的影响。此外，我们收集了一个标注良好的风格数据集Style30k，用于训练一个风格特征提取器，该提取器能够准确表示风格，同时排除其他内容信息。实验结果表明，我们的方法充分掌握了参考图像的风格特征，生成了既与目标图像风格一致又与文本提示一致的吸引人的图像。代码和数据集可在 https://github.com/alipay/style-tokenizer 获取。
英文摘要：
Despite the burst of innovative methods for controlling the diffusion process, effectively controlling image styles in text-to-image generation remains a challenging task. Many adapter-based methods impose image representation conditions on the denoising process to accomplish image control. However these conditions are not aligned with the word embedding space, leading to interference between image and text control conditions and the potential loss of semantic information from the text prompt. Addressing this issue involves two key challenges. Firstly, how to inject the style representation without compromising the effectiveness of text representation in control. Secondly, how to obtain the accurate style representation from a single reference image. To tackle these challenges, we introduce StyleTokenizer, a zero-shot style control image generation method that aligns style representation with text representation using a style tokenizer. This alignment effectively minimizes the impact on the effectiveness of text prompts. Furthermore, we collect a well-labeled style dataset named Style30k to train a style feature extractor capable of accurately representing style while excluding other content information. Experimental results demonstrate that our method fully grasps the style characteristics of the reference image, generating appealing images that are consistent with both the target image style and text prompt. The code and dataset are available at https://github.com/alipay/style-tokenizer.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
