arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2406.06525cs.CV
收藏

自回归模型超越扩散模型:Llama 实现可扩展图像生成

Autoregressive Model Beats Diffusion: Llama for Scalable Image Generation

The University of Hong Kong(香港大学);ByteDance(字节跳动)

Peize Sun, Yi Jiang, Shoufa Chen, Shilong Zhang, Bingyue Peng, Ping Luo, Zehuan Yuan

AI总结:

本文提出 LlamaGen,将 Llama 式自回归下一 token 预测范式应用于图像生成,通过优化图像分词器、模型扩展和训练数据,在 ImageNet 上以 2.18 FID 超越扩散模型,并验证了 LLM 服务框架可带来 326%-414% 的推理加速。

摘要:

我们提出 LlamaGen,这是一个新的图像生成模型家族,将大语言模型原有的“下一 token 预测”范式应用于视觉生成领域。它肯定地回答了一个问题:如果没有针对视觉信号的归纳偏置,像 Llama 这样的普通自回归模型在适当扩展的情况下,是否能够实现最先进的图像生成性能。我们重新审视了图像分词器的设计空间、图像生成模型的可扩展性特性及其训练数据质量。这一探索的成果包括:(1)一个下采样比例为 16 的图像分词器,在 ImageNet 基准上重建质量为 0.94 rFID,码本利用率为 97%。(2)一系列参数规模从 111M 到 3.1B 的类别条件图像生成模型,在 ImageNet 256x256 基准上达到 2.18 FID,优于 LDM、DiT 等流行的扩散模型。(3)一个具有 775M 参数的文本条件图像生成模型,通过在 LAION-COCO 和高美学质量图像上进行两阶段训练,展现出具有竞争力的视觉质量和文本对齐性能。(4)我们验证了 LLM 服务框架在优化图像生成模型推理速度方面的有效性,实现了 326% - 414% 的加速。我们发布所有模型和代码,以促进视觉生成和多模态基础模型的开源社区发展。

英文摘要:

We introduce LlamaGen, a new family of image generation models that apply original ``next-token prediction'' paradigm of large language models to visual generation domain. It is an affirmative answer to whether vanilla autoregressive models, e.g., Llama, without inductive biases on visual signals can achieve state-of-the-art image generation performance if scaling properly. We reexamine design spaces of image tokenizers, scalability properties of image generation models, and their training data quality. The outcome of this exploration consists of: (1) An image tokenizer with downsample ratio of 16, reconstruction quality of 0.94 rFID and codebook usage of 97% on ImageNet benchmark. (2) A series of class-conditional image generation models ranging from 111M to 3.1B parameters, achieving 2.18 FID on ImageNet 256x256 benchmarks, outperforming the popular diffusion models such as LDM, DiT. (3) A text-conditional image generation model with 775M parameters, from two-stage training on LAION-COCO and high aesthetics quality images, demonstrating competitive performance of visual quality and text alignment. (4) We verify the effectiveness of LLM serving frameworks in optimizing the inference speed of image generation models and achieve 326% - 414% speedup. We release all models and codes to facilitate open-source community of visual generation and multimodal foundation models.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑