通过从粗到细的Token预测改进自回归图像生成
Improving Autoregressive Image Generation through Coarse-to-Fine Token Prediction
AI总结：
针对自回归图像生成中大码本会增加建模难度的问题，提出从粗到细的token预测框架，在ImageNet上Inception Score较基线平均提升59分，且采样速度更快。
摘要：
自回归模型通过借鉴语言建模中的序列预测技术，已在图像生成领域取得了显著成效。然而，将这类方法应用于图像任务时，需要通过VQ-VAE（矢量量化变分自编码器）等矢量量化方法对连续像素数据进行离散化处理。为缓解VQ-VAE中存在的量化误差，近期研究倾向于使用更大的码本，但这会相应扩大词表规模，增加自回归建模任务的复杂度。本文旨在找到一种既能利用大码本优势、又不会提升自回归建模难度的方法。通过实证研究我们发现，具有相似码字表示的token会对最终生成图像产生相似效果，这表明大码本中存在显著冗余。基于这一洞见，我们提出从粗到细（CTF）的token预测方法，通过为相似token分配相同的粗标签来实现。我们的框架包含两个阶段：（1）自回归模型，依次为序列中的每个token预测粗标签；（2）辅助模型，以粗标签为条件，同时预测所有token的细粒度标签。在ImageNet上的实验证明了我们方法的优越性能，与基线相比，Inception Score平均提升59分。值得注意的是，尽管增加了一个推理步骤，我们的方法仍实现了更快的采样速度。
英文摘要：
Autoregressive models have shown remarkable success in image generation by adapting sequential prediction techniques from language modeling. However, applying these approaches to images requires discretizing continuous pixel data through vector quantization methods like VQ-VAE. To alleviate the quantization errors that existed in VQ-VAE, recent works tend to use larger codebooks. However, this will accordingly expand vocabulary size, complicating the autoregressive modeling task. This paper aims to find a way to enjoy the benefits of large codebooks without making autoregressive modeling more difficult. Through empirical investigation, we discover that tokens with similar codeword representations produce similar effects on the final generated image, revealing significant redundancy in large codebooks. Based on this insight, we propose to predict tokens from coarse to fine (CTF), realized by assigning the same coarse label for similar tokens. Our framework consists of two stages: (1) an autoregressive model that sequentially predicts coarse labels for each token in the sequence, and (2) an auxiliary model that simultaneously predicts fine-grained labels for all tokens conditioned on their coarse labels. Experiments on ImageNet demonstrate our method's superior performance, achieving an average improvement of 59 points in Inception Score compared to baselines. Notably, despite adding an inference step, our approach achieves faster sampling speeds.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
