arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2210.08933cs.CLcs.LG
收藏

DiffuSeq:基于扩散模型的序列到序列文本生成

DiffuSeq: Sequence to Sequence Text Generation with Diffusion Models

Shanghai AI Laboratory(上海人工智能实验室);The University of Hong Kong(香港大学)

Shansan Gong, Mukai Li, Jiangtao Feng, Zhiyong Wu, Lingpeng Kong

摘要:

最近,扩散模型已经成为生成模型的一个新的范例。尽管在使用连续信号(如视觉和音频)的领域取得了成功,但由于文本的离散性,特别是对于条件生成,使扩散模型适应自然语言的研究还不够。我们通过提出DiffuSeq来应对这一挑战:一个为序列到序列(Seq 2Seq)文本生成任务设计的扩散模型。在对各种Seq 2Seq任务进行广泛评估后,我们发现DiffuSeq实现了与六个已建立的基线相当甚至更好的性能,包括基于预训练语言模型的最先进模型。除了质量之外,DiffuSeq的一个有趣特性是其在生成过程中的高多样性,这是许多Seq 2Seq任务所期望的。我们还包括一个理论分析,揭示了DiffuSeq和自回归/非自回归模型之间的联系。结合理论分析和经验证据,我们证明了扩散模型在复杂的条件语言生成任务的巨大潜力。代码可在\url{https://github.com/Shark-NLP/DiffuSeq}上获得

英文摘要:

Recently, diffusion models have emerged as a new paradigm for generative models. Despite the success in domains using continuous signals such as vision and audio, adapting diffusion models to natural language is under-explored due to the discrete nature of texts, especially for conditional generation. We tackle this challenge by proposing DiffuSeq: a diffusion model designed for sequence-to-sequence (Seq2Seq) text generation tasks. Upon extensive evaluation over a wide range of Seq2Seq tasks, we find DiffuSeq achieving comparable or even better performance than six established baselines, including a state-of-the-art model that is based on pre-trained language models. Apart from quality, an intriguing property of DiffuSeq is its high diversity during generation, which is desired in many Seq2Seq tasks. We further include a theoretical analysis revealing the connection between DiffuSeq and autoregressive/non-autoregressive models. Bringing together theoretical analysis and empirical evidence, we demonstrate the great potential of diffusion models in complex conditional language generation tasks. Code is available at \url{https://github.com/Shark-NLP/DiffuSeq}

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑