StyleAR：定制多模态自回归模型用于风格对齐的文本到图像生成
StyleAR: Customizing Multimodal Autoregressive Model for Style-Aligned Text-to-Image Generation
AI总结：
StyleAR通过专门的数据整理方法利用文本-图像二元数据，结合CLIP编码器和风格增强令牌，实现风格对齐的文本到图像生成，并取得优越性能。
摘要：
在当前的研究格局中，多模态自回归（AR）模型在包括视觉理解和生成在内的各个领域展现出卓越的能力。然而，诸如风格对齐的文本到图像生成等复杂任务仍面临重大挑战，尤其是在数据获取方面。类比于AR模型在图像编辑中的指令跟随调优，风格对齐生成需要参考风格图像和提示词，从而形成文本-图像到图像的三元组，其中输出与输入共享风格和语义。然而，获取大量具有特定风格的此类三元组数据比获取用于训练生成模型的传统文本到图像数据要困难得多。为解决此问题，我们提出StyleAR，一种创新方法，结合专门设计的数据整理方法与我们所提出的AR模型，以有效利用文本到图像二元数据实现风格对齐的文本到图像生成。我们的方法使用参考风格图像和提示词合成目标风格化数据，但仅将目标风格化图像作为图像模态来创建高质量的二元数据。为促进二元数据训练，我们引入带有感知器重采样器的CLIP图像编码器，将图像输入转换为与AR模型中的多模态令牌对齐的风格令牌，并实现风格增强令牌技术以防止先前工作中常见的内容泄漏问题。此外，我们将从大规模文本-图像数据集中提取的原始图像与风格化图像混合，以增强StyleAR提取更丰富风格特征并确保风格一致性的能力。广泛的定性和定量实验证明了我们方法的优越性能。
英文摘要：
In the current research landscape, multimodal autoregressive (AR) models have shown exceptional capabilities across various domains, including visual understanding and generation. However, complex tasks such as style-aligned text-to-image generation present significant challenges, particularly in data acquisition. In analogy to instruction-following tuning for image editing of AR models, style-aligned generation requires a reference style image and prompt, resulting in a text-image-to-image triplet where the output shares the style and semantics of the input. However, acquiring large volumes of such triplet data with specific styles is considerably more challenging than obtaining conventional text-to-image data used for training generative models. To address this issue, we propose StyleAR, an innovative approach that combines a specially designed data curation method with our proposed AR models to effectively utilize text-to-image binary data for style-aligned text-to-image generation. Our method synthesizes target stylized data using a reference style image and prompt, but only incorporates the target stylized image as the image modality to create high-quality binary data. To facilitate binary data training, we introduce a CLIP image encoder with a perceiver resampler that translates the image input into style tokens aligned with multimodal tokens in AR models and implement a style-enhanced token technique to prevent content leakage which is a common issue in previous work. Furthermore, we mix raw images drawn from large-scale text-image datasets with stylized images to enhance StyleAR's ability to extract richer stylistic features and ensure style consistency. Extensive qualitative and quantitative experiments demonstrate our superior performance.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
