arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2303.06574cs.CL
收藏

用于非自回归文本生成的扩散模型:综述

Diffusion Models for Non-autoregressive Text Generation: A Survey

Gaoling School of Artificial Intelligence, Renmin University of China(中国人民大学高瓴人工智能学院);School of Information, Renmin University of China(中国人民大学信息学院);Beijing Key Laboratory of Big Data Management and Analysis Methods(北京大数据管理与分析方法重点实验室)

Yifan Li, Kun Zhou, Wayne Xin Zhao, Ji-Rong Wen

AI总结:

本综述系统梳理了扩散模型在非自回归文本生成中的应用,涵盖模型定义、主流方法、关键设计、预训练语言模型利用及优化技术,并展望了未来方向。

摘要:

非自回归(NAR)文本生成在自然语言处理领域引起了广泛关注,它大幅降低了推理延迟,但不得不牺牲生成准确性。近年来,扩散模型作为一类潜在变量生成模型,被引入到NAR文本生成中,显示出改进的文本生成质量。在本综述中,我们回顾了用于NAR文本生成的扩散模型的最新进展。作为背景,我们首先介绍扩散模型和文本扩散模型的一般定义,然后讨论它们在NAR生成中的优势。作为核心内容,我们进一步介绍现有文本扩散工作中的两种主流扩散模型,并回顾扩散过程的关键设计。此外,我们讨论了预训练语言模型(PLMs)在文本扩散模型中的利用,并介绍了针对文本数据的优化技术。最后,我们讨论了几个有前景的研究方向并总结本文。本综述旨在为研究人员提供关于用于NAR生成的文本扩散模型相关研究的系统性参考。我们在https://github.com/RUCAIBox/Awesome-Text-Diffusion-Models上提供了我们收集的文本扩散模型列表。

英文摘要:

Non-autoregressive (NAR) text generation has attracted much attention in the field of natural language processing, which greatly reduces the inference latency but has to sacrifice the generation accuracy. Recently, diffusion models, a class of latent variable generative models, have been introduced into NAR text generation, showing an improved text generation quality. In this survey, we review the recent progress in diffusion models for NAR text generation. As the background, we first present the general definition of diffusion models and the text diffusion models, and then discuss their merits for NAR generation. As the core content, we further introduce two mainstream diffusion models in existing work of text diffusion, and review the key designs of the diffusion process. Moreover, we discuss the utilization of pre-trained language models (PLMs) for text diffusion models and introduce optimization techniques for text data. Finally, we discuss several promising directions and conclude this paper. Our survey aims to provide researchers with a systematic reference of related research on text diffusion models for NAR generation. We present our collection of text diffusion models at https://github.com/RUCAIBox/Awesome-Text-Diffusion-Models.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑