LANTERN：通过松弛投机解码加速视觉自回归模型
LANTERN: Accelerating Visual Autoregressive Models with Relaxed Speculative Decoding
AI总结：
针对视觉自回归模型因token选择歧义导致投机解码加速受限的问题，提出松弛接受条件LANTERN，利用潜空间token可互换性提升解码效率，在保证图像质量的同时实现1.75倍至1.82倍的加速。
摘要：
自回归（AR）模型近期在图像生成领域崭露头角，其性能常常匹敌甚至超越扩散模型。然而，AR模型的一个主要局限在于其序列特性，即一次处理一个token，这使得其生成速度慢于GAN或扩散方法等高效运行的模型。尽管投机解码已被证明能通过单次前向传播生成多个token来加速LLM，但其在视觉AR模型中的应用仍鲜有探索。在本研究中，我们识别了该场景中的一项挑战，我们将其称为“token选择歧义”，即视觉AR模型经常为token分配均匀的低概率，从而阻碍了投机解码的性能。为克服这一挑战，我们提出了一种松弛接受条件，称为LANTERN，它利用了token在潜空间中的可互换性。这种松弛通过更灵活地使用原本会被过早拒绝的候选token，恢复了投机解码在视觉AR模型中的有效性。此外，通过引入总变差距离界限，我们确保在不显著损害图像质量或语义连贯性的前提下实现这些速度提升。实验结果证明了我们方法在提供显著加速方面的有效性。具体而言，与直接应用最先进的投机解码相比，当应用于当代视觉AR模型LlamaGen时，LANTERN相比于贪心解码和随机采样，分别将速度提升了1.75倍和1.82倍。代码已在 https://github.com/jadohu/LANTERN 公开。
英文摘要：
Auto-Regressive (AR) models have recently gained prominence in image generation, often matching or even surpassing the performance of diffusion models. However, one major limitation of AR models is their sequential nature, which processes tokens one at a time, slowing down generation compared to models like GANs or diffusion-based methods that operate more efficiently. While speculative decoding has proven effective for accelerating LLMs by generating multiple tokens in a single forward, its application in visual AR models remains largely unexplored. In this work, we identify a challenge in this setting, which we term \textit{token selection ambiguity}, wherein visual AR models frequently assign uniformly low probabilities to tokens, hampering the performance of speculative decoding. To overcome this challenge, we propose a relaxed acceptance condition referred to as LANTERN that leverages the interchangeability of tokens in latent space. This relaxation restores the effectiveness of speculative decoding in visual AR models by enabling more flexible use of candidate tokens that would otherwise be prematurely rejected. Furthermore, by incorporating a total variation distance bound, we ensure that these speed gains are achieved without significantly compromising image quality or semantic coherence. Experimental results demonstrate the efficacy of our method in providing a substantial speed-up over speculative decoding. In specific, compared to a naïve application of the state-of-the-art speculative decoding, LANTERN increases speed-ups by $\mathbf{1.75}\times$ and $\mathbf{1.82}\times$, as compared to greedy decoding and random sampling, respectively, when applied to LlamaGen, a contemporary visual AR model. The code is publicly available at https://github.com/jadohu/LANTERN.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
