arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2412.17153cs.CVcs.LG
收藏

蒸馏解码1:利用流匹配实现图像自回归模型的一步采样

Distilled Decoding 1: One-step Sampling of Image Auto-regressive Models with Flow Matching

Tsinghua University(清华大学);Microsoft Research(微软研究院)

Enshu Liu, Xuefei Ning, Yu Wang, Zinan Lin

AI总结:

本文提出蒸馏解码(DD),利用流匹配将高斯分布映射到预训练图像自回归模型的输出分布,实现一步或少步生成,在VAR和LlamaGen上分别获得6.3倍和217.8倍加速,同时保持可接受的FID,挑战了AR模型速度慢的固有观念。

摘要:

自回归(AR)模型在文本和图像生成中已达到最先进的性能,但由于逐token生成的过程,其生成速度较慢。我们提出了一个雄心勃勃的问题:能否将预训练的AR模型调整为仅需一步或两步即可生成输出?如果成功,这将显著推进AR模型的发展与部署。我们注意到,现有试图通过一次生成多个token来加速AR生成的工作,由于token之间的条件依赖性,从根本上无法捕获输出分布,限制了其在少步生成中的有效性。为解决此问题,我们提出了蒸馏解码(DD),该方法利用流匹配建立从高斯分布到预训练AR模型输出分布的确定性映射。随后,我们训练一个网络来蒸馏该映射,从而实现少步生成。DD不需要原始AR模型的训练数据,使其更具实用性。我们在最先进的图像AR模型上评估了DD,并在ImageNet-256上展示了有前景的结果。对于需要10步生成的VAR,DD实现了一步生成(6.3倍加速),FID从4.19增加到9.96,增幅可接受。对于LlamaGen,DD将生成步骤从256步减少到1步,实现了217.8倍加速,FID从4.11增加到11.35,增幅相当。在两种情况下,基线方法完全失败,FID大于100。DD在文本到图像生成方面也表现出色,将LlamaGen的生成步骤从256步减少到2步,FID从25.70增加到28.95,增幅极小。作为首个证明图像AR模型一步生成可能性的工作,DD挑战了AR模型本质上缓慢的普遍观念,并为高效AR生成开辟了新机遇。项目网站为https://imagination-research.github.io/distilled-decoding。

英文摘要:

Autoregressive (AR) models have achieved state-of-the-art performance in text and image generation but suffer from slow generation due to the token-by-token process. We ask an ambitious question: can a pre-trained AR model be adapted to generate outputs in just one or two steps? If successful, this would significantly advance the development and deployment of AR models. We notice that existing works that try to speed up AR generation by generating multiple tokens at once fundamentally cannot capture the output distribution due to the conditional dependencies between tokens, limiting their effectiveness for few-step generation. To address this, we propose Distilled Decoding (DD), which uses flow matching to create a deterministic mapping from Gaussian distribution to the output distribution of the pre-trained AR model. We then train a network to distill this mapping, enabling few-step generation. DD doesn't need the training data of the original AR model, making it more practical. We evaluate DD on state-of-the-art image AR models and present promising results on ImageNet-256. For VAR, which requires 10-step generation, DD enables one-step generation (6.3$\times$ speed-up), with an acceptable increase in FID from 4.19 to 9.96. For LlamaGen, DD reduces generation from 256 steps to 1, achieving an 217.8$\times$ speed-up with a comparable FID increase from 4.11 to 11.35. In both cases, baseline methods completely fail with FID>100. DD also excels on text-to-image generation, reducing the generation from 256 steps to 2 for LlamaGen with minimal FID increase from 25.70 to 28.95. As the first work to demonstrate the possibility of one-step generation for image AR models, DD challenges the prevailing notion that AR models are inherently slow, and opens up new opportunities for efficient AR generation. The project website is at https://imagination-research.github.io/distilled-decoding.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑