Diffusion Models for Non-autoregressive Text Generation: A Survey
Gaoling School of Artificial Intelligence, Renmin University of China(中国人民大学高瓴人工智能学院);School of Information, Renmin University of China(中国人民大学信息学院);Beijing Key Laboratory of Big Data Management and Analysis Methods(北京大数据管理与分析方法重点实验室)
Non-autoregressive (NAR) text generation has attracted much attention in the field of natural language processing, which greatly reduces the inference latency but has to sacrifice the generation accuracy. Recently, diffusion models, a class of latent variable generative models, have been introduced into NAR text generation, showing an improved text generation quality. In this survey, we review the recent progress in diffusion models for NAR text generation. As the background, we first present the general definition of diffusion models and the text diffusion models, and then discuss their merits for NAR generation. As the core content, we further introduce two mainstream diffusion models in existing work of text diffusion, and review the key designs of the diffusion process. Moreover, we discuss the utilization of pre-trained language models (PLMs) for text diffusion models and introduce optimization techniques for text data. Finally, we discuss several promising directions and conclude this paper. Our survey aims to provide researchers with a systematic reference of related research on text diffusion models for NAR generation. We present our collection of text diffusion models at https://github.com/RUCAIBox/Awesome-Text-Diffusion-Models.
Hierarchical Skip Decoding for Efficient Autoregressive Text Generation
School of Information and Communication Engineering, Hainan University(海南大学信息与通信工程学院);Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所);Guangzhou University(广州大学)
Differences in Text Generated by Diffusion and Autoregressive Language Models
Shanghai Qi Zhi Institute(上海启智研究院);Institute for Interdisciplinary Information Sciences, Tsinghua University(清华大学交叉信息研究院);Xiongan AI Institute(雄安人工智能研究院)
Diffusion vs. Autoregressive Language Models: A Text Embedding Perspective
Nanyang Technological University(南洋理工大学);Yale University(耶鲁大学);NYU Shanghai(纽约大学上海校区);Alibaba-NTU Singapore Joint Research Institute(阿里巴巴-国立科技大学新加坡联合研究机构);Center for Data Science New York University(纽约大学数据科学中心)
Autoregressive Models Rival Diffusion Models at ANY-ORDER Generation
National Key Lab of General Artificial Intelligence, School of Intelligence Science and Technology, Peking University(国家关键人工智能实验室,智能科学与技术学院,北京大学);School of Mathematical Sciences, Peking University(数学科学学院,北京大学);Amazon AGI SF Lab(亚马逊人工智能SF实验室);Institute for Artificial Intelligence, Peking University(人工智能研究院,北京大学)
Mechanism Shift During Post-training from Autoregressive to Masked Diffusion Language Models
Graduate School of Data Science, Seoul National University(首尔国立大学数据科学研究生院);Department of Biosystems & Biomaterials Science and Engineering, Seoul National University(首尔国立大学生物系统与生物材料科学与工程系)
Analyzing Diffusion and Autoregressive Vision Language Models in Multimodal Embedding Space
Nanyang Technological University(南洋理工大学);Yale University(耶鲁大学);NYU Shanghai(纽约大学上海分校);Alibaba-NTU Singapore Joint Research Institute(阿里-国立新加坡大学联合研究机构);University of the Chinese Academy of Sciences(中国科学院大学);Center for Data Science(数据科学中心);New York University(纽约大学)
Why Diffusion Language Models Struggle with Truly Parallel (Non-Autoregressive) Decoding?
The Hong Kong Polytechnic University, Hong Kong, China(香港理工大学);ELLIS Institute Tübingen, Tübingen, Germany(图宾根ELLIS研究所);Max Planck Institute for Intelligent Systems, Tübingen, Germany(智能系统马克斯·普朗克研究所);Tübingen AI Center, Tübingen, Germany(图宾根人工智能中心);University of Surrey, Guildford, United Kingdom(萨里大学);The University of North Carolina at Chapel Hill, Chapel Hill, NC, USA(北卡罗来纳大学教堂山分校)
Talker-T2AV: Joint Talking Audio-Video Generation with Autoregressive Diffusion Modeling
Hong Kong University of Science and Technology(香港科学与技术大学);Zhejiang University(浙江大学);National University of Singapore(新加坡国立大学);The Chinese University of Hong Kong(香港中文大学);Peking University(北京大学);Independent Researcher(独立研究者)