Autoregressive (AR) models have achieved state-of-the-art performance in text and image generation but suffer from slow generation due to the token-by-token process. We ask an ambitious question: can a pre-trained AR model be adapted to generate outputs in just one or two steps? If successful, this would significantly advance the development and deployment of AR models. We notice that existing works that try to speed up AR generation by generating multiple tokens at once fundamentally cannot capture the output distribution due to the conditional dependencies between tokens, limiting their effectiveness for few-step generation. To address this, we propose Distilled Decoding (DD), which uses flow matching to create a deterministic mapping from Gaussian distribution to the output distribution of the pre-trained AR model. We then train a network to distill this mapping, enabling few-step generation. DD doesn't need the training data of the original AR model, making it more practical. We evaluate DD on state-of-the-art image AR models and present promising results on ImageNet-256. For VAR, which requires 10-step generation, DD enables one-step generation (6.3$\times$ speed-up), with an acceptable increase in FID from 4.19 to 9.96. For LlamaGen, DD reduces generation from 256 steps to 1, achieving an 217.8$\times$ speed-up with a comparable FID increase from 4.11 to 11.35. In both cases, baseline methods completely fail with FID>100. DD also excels on text-to-image generation, reducing the generation from 256 steps to 2 for LlamaGen with minimal FID increase from 25.70 to 28.95. As the first work to demonstrate the possibility of one-step generation for image AR models, DD challenges the prevailing notion that AR models are inherently slow, and opens up new opportunities for efficient AR generation. The project website is at https://imagination-research.github.io/distilled-decoding.
Accelerating Auto-regressive Text-to-Image Generation with Training-free Speculative Jacobi Decoding
The University of Hong Kong(香港大学);Huawei Noah’s Ark Lab(华为诺亚方舟实验室);CUHK(香港中文大学);Tsinghua University(清华大学);Shanghai Jiao Tong University(上海交通大学);Infinigence AI(智源人工智能)
Pushing Auto-regressive Models for 3D Shape Generation at Capacity and Scalability
Fudan University(复旦大学);Fudan ISTBI—ZJNU Algorithm Centre for Brain-inspired Intelligence(复旦大学ISTBI—浙江师范大学类脑智能算法中心);Tsinghua University(清华大学);Google(谷歌公司);Nanjing University(南京大学);Tencent Youtu lab(腾讯优图实验室);BAAI (Beijing Academy of Artificial Intelligence)(北京智源人工智能研究院)
X-Prompt: Towards Universal In-Context Image Generation in Auto-Regressive Vision Language Foundation Models
Shanghai Jiao Tong University(上海交通大学);Shanghai AI Laboratory(上海人工智能实验室);Tsinghua University(清华大学);The Chinese University of Hong Kong(香港中文大学);CPII under InnoHK(创新香港科技促进会);MThreads AI
SJD++: Improved Speculative Jacobi Decoding for Training-free Acceleration of Discrete Auto-regressive Text-to-Image Generation
The University of Hong Kong(香港大学);Huawei Noah’s Ark Lab(华为诺亚实验室);Peking University(北京大学);xAI;Tsinghua University(清华大学);Shanghai Jiao Tong University(上海交通大学)
Institute of Medical Technology, Peking University Health Science Center, Peking University(北京大学医学部医学技术研究所);Baidu VIS(百度VIS);Biomedical Engineering Department, College of Future Technology, Peking University(未来技术学院生物医学工程系);National Biomedical Imaging Center, Peking University(北京大学国家生物医学成像中心);Tsinghua University(清华大学)
Error Analyses of Auto-Regressive Video Diffusion Models: A Unified Framework
Nanyang Technological University(南洋理工大学);National University of Singapore(国立新加坡大学);Singapore University of Technology and Design(新加坡科技设计大学);Sea AI Lab(海思人工智能实验室);Yale University(耶鲁大学)
TNPAR: Topological Neural Poisson Auto-Regressive Model for Learning Granger Causal Structure from Event Sequences
Guangdong University of Technology(广东工业大学);Peng Cheng Laboratory(鹏城实验室);Mohamed bin Zayed University of Artificial Intelligence(穆罕默德·本·扎耶德人工智能大学);Huawei(华为);Shantou University(汕头大学)
XSpecMesh: Quality-Preserving Auto-Regressive Mesh Generation Acceleration via Multi-Head Speculative Decoding
Key Laboratory of Multimedia Trusted Perception and Efficient Computing, Ministry of Education of China, Xiamen University(教育部多媒体可信感知与高效计算重点实验室,厦门大学);Shandong Inspur Database Technology Co.,Ltd(山东浪潮数据库技术有限公司)
Multi-modal Auto-regressive Modeling via Visual Words
School of Computer Science, Wuhan University(武汉大学计算机学院);Shanghai Jiao Tong University(上海交通大学);School of Information Management, Wuhan University(武汉大学信息管理学院)
High-Fidelity Functional Ultrasound Reconstruction via A Visual Auto-Regressive Framework
Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences(深圳先进技术研究院,中国科学院);Department of Computer and Information Science, University of Macau(计算机与信息科学系,澳门大学)
Quality-Aware Language-Conditioned Local Auto-Regressive Anomaly Synthesis and Detection
Foundation Model Research Center, Institute of Automation, Chinese Academy of Sciences(自动化研究所基础模型研究中心,中国科学院);School of Artificial Intelligence, University of Chinese Academy of Sciences(中国科学院大学人工智能学院);Objecteye Inc.(Objecteye公司)