arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2506.09482cs.CV
收藏

将自回归Transformer与扩散模型通过多参考自回归相结合

Marrying Autoregressive Transformer and Diffusion with Multi-Reference Autoregression

Soul AI;ICT;Chinese Academy of Sciences(中国科学院)

Dingcheng Zhen, Qian Qiao, Xu Zheng, Tan Yu, Kangxi Wu, Ziwei Zhang, Siyuan Liu, Shunshun Yin, Ming Tao

AI总结:

该研究提出首个结合自回归Transformer与扩散模型的图像生成模型TransDiff,其在ImageNet 256x256基准上FID达1.61、IS达293.4且推理速度快,还基于其提出MRAR范式进一步提升性能,为图像生成开辟新方向。

摘要:

我们提出TransDiff,这是首个将自回归(AR)Transformer与扩散模型相结合的图像生成模型。在该联合建模框架中,TransDiff将标签和图像编码为高级语义特征,并采用扩散模型估计图像样本的分布。在ImageNet 256x256基准测试中,TransDiff显著优于基于独立AR Transformer或扩散模型的其他图像生成模型。具体而言,TransDiff实现了1.61的弗雷歇 inception 距离(FID)和293.4的inception得分(IS),并且与基于AR Transformer的最先进方法相比,推理延迟快2倍,与纯扩散模型相比快112倍。此外,基于TransDiff模型,我们引入了一种名为多参考自回归(MRAR)的新型图像生成范式,该范式通过预测下一张图像来执行自回归生成。MRAR使模型能够参考多个先前生成的图像,从而促进学习更多样化的表示,并提高后续迭代中生成图像的质量。通过应用MRAR,TransDiff的性能得到提升,FID从1.61降至1.42。我们期望TransDiff能为图像生成领域开辟新前沿。

英文摘要:

We introduce TransDiff, the first image generation model that marries Autoregressive (AR) Transformer with diffusion models. In this joint modeling framework, TransDiff encodes labels and images into high-level semantic features and employs a diffusion model to estimate the distribution of image samples. On the ImageNet 256x256 benchmark, TransDiff significantly outperforms other image generation models based on standalone AR Transformer or diffusion models. Specifically, TransDiff achieves a Frechet Inception Distance (FID) of 1.61 and an Inception Score (IS) of 293.4, and further provides x2 faster inference latency compared to state-of-the-art methods based on AR Transformer and x112 faster inference compared to diffusion-only models. Furthermore, building on the TransDiff model, we introduce a novel image generation paradigm called Multi-Reference Autoregression (MRAR), which performs autoregressive generation by predicting the next image. MRAR enables the model to reference multiple previously generated images, thereby facilitating the learning of more diverse representations and improving the quality of generated images in subsequent iterations. By applying MRAR, the performance of TransDiff is improved, with the FID reduced from 1.61 to 1.42. We expect TransDiff to open up a new frontier in the field of image generation.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑