使用免训练的投机Jacobi解码加速自回归文本到图像生成
Accelerating Auto-regressive Text-to-Image Generation with Training-free Speculative Jacobi Decoding
AI总结：
本文提出了一种免训练的概率并行解码算法SJD，通过引入概率收敛标准在保持生成多样性的同时，有效加速了自回归文本到图像生成。
摘要：
当前的大型自回归模型能够生成高质量、高分辨率的图像，但这些模型在推理过程中需要数百甚至数千步的下一个token预测，导致大量时间消耗。在现有研究中，Jacobi decoding（一种迭代并行解码算法）已被用于加速自回归生成，并且可以在无需训练的情况下执行。然而，Jacobi decoding依赖确定性标准来判断迭代的收敛性。因此，它适用于贪心解码，但与基于采样的解码不兼容，而后者对于当前自回归文本到图像生成中的视觉质量和多样性至关重要。在本文中，我们提出了一种免训练的概率并行解码算法Speculative Jacobi Decoding (SJD)，以加速自回归文本到图像生成。通过引入概率收敛标准，我们的SJD在加速自回归文本到图像生成推理的同时，保持了基于采样的token解码中的随机性，并允许模型生成多样化的图像。具体而言，SJD促使模型在每一步预测多个token，并基于概率标准接受token，使得模型能够以比传统的下一个token预测范式更少的步骤生成图像。我们还研究了利用视觉数据空间局部性的token初始化策略，以进一步提高特定场景下的加速比。我们在多个自回归文本到图像生成模型上对提出的SJD进行了实验，证明了在不牺牲视觉质量的情况下模型加速的有效性。我们的代码可在此处获取：https://github.com/tyshiwo1/Accelerating-T2I-AR-with-SJD/。
英文摘要：
The current large auto-regressive models can generate high-quality, high-resolution images, but these models require hundreds or even thousands of steps of next-token prediction during inference, resulting in substantial time consumption. In existing studies, Jacobi decoding, an iterative parallel decoding algorithm, has been used to accelerate the auto-regressive generation and can be executed without training. However, the Jacobi decoding relies on a deterministic criterion to determine the convergence of iterations. Thus, it works for greedy decoding but is incompatible with sampling-based decoding which is crucial for visual quality and diversity in the current auto-regressive text-to-image generation. In this paper, we propose a training-free probabilistic parallel decoding algorithm, Speculative Jacobi Decoding (SJD), to accelerate auto-regressive text-to-image generation. By introducing a probabilistic convergence criterion, our SJD accelerates the inference of auto-regressive text-to-image generation while maintaining the randomness in sampling-based token decoding and allowing the model to generate diverse images. Specifically, SJD facilitates the model to predict multiple tokens at each step and accepts tokens based on the probabilistic criterion, enabling the model to generate images with fewer steps than the conventional next-token-prediction paradigm. We also investigate the token initialization strategies that leverage the spatial locality of visual data to further improve the acceleration ratio under specific scenarios. We conduct experiments for our proposed SJD on multiple auto-regressive text-to-image generation models, showing the effectiveness of model acceleration without sacrificing the visual quality. The code of our work is available here: https://github.com/tyshiwo1/Accelerating-T2I-AR-with-SJD/.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
