arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2305.09515cs.CL
收藏

AR-Diffusion:用于文本生成的自回归扩散模型

AR-Diffusion: Auto-Regressive Diffusion Model for Text Generation

Microsoft Research Asia(微软亚洲研究院);Microsoft Azure AI(微软Azure人工智能);Microsoft(微软);Soochow University(苏州大学);Fudan University(复旦大学);IDEA Research(IDEA研究院)

Tong Wu, Zhihao Fan, Xiao Liu, Yeyun Gong, Yelong Shen, Jian Jiao, Hai-Tao Zheng, Juntao Li, Zhongyu Wei, Jian Guo, Nan Duan, Weizhu Chen

AI总结:

针对文本扩散模型忽视语言序列依赖的问题,提出按 token 位置动态分配去噪步数的 AR-Diffusion,实现从左到右生成,并在多任务上以最高约600倍加速超越现有扩散语言模型。

摘要:

扩散模型因其卓越性能在图像生成领域获得了广泛关注。最近,其成功已通过同时生成一个序列中的所有 token 而扩展到文本生成。然而,与图像相比,自然语言表现出明显更强的序列依赖,而且大多数现有语言模型都采用从左到右的自回归方式训练。为考虑自然语言固有的序列特性,我们提出了 Auto-Regressive Diffusion(AR-Diffusion,自回归扩散)。AR-Diffusion 确保右侧 token 的生成依赖于左侧已生成的 token,该机制通过使用随 token 位置变化的动态去噪步数实现。这使得左侧 token 经历的去噪步数少于右侧 token,从而使它们能够更早生成,并随后影响右侧 token 的生成。在包括文本摘要、机器翻译和常识生成在内的多种文本生成任务的一系列实验中,AR-Diffusion 清楚展示了其相对于现有扩散语言模型的优越性,并且在取得可比结果时速度可快 100×~600×。我们的代码可在 https://github.com/microsoft/ProphetNet/tree/master/AR-diffusion 获取。

英文摘要:

Diffusion models have gained significant attention in the realm of image generation due to their exceptional performance. Their success has been recently expanded to text generation via generating all tokens within a sequence concurrently. However, natural language exhibits a far more pronounced sequential dependency in comparison to images, and the majority of existing language models are trained with a left-to-right auto-regressive approach. To account for the inherent sequential characteristic of natural language, we introduce Auto-Regressive Diffusion (AR-Diffusion). AR-Diffusion ensures that the generation of tokens on the right depends on the generated ones on the left, a mechanism achieved through employing a dynamic number of denoising steps that vary based on token position. This results in tokens on the left undergoing fewer denoising steps than those on the right, thereby enabling them to generate earlier and subsequently influence the generation of tokens on the right. In a series of experiments on various text generation tasks, including text summarization, machine translation, and common sense generation, AR-Diffusion clearly demonstrated its superiority over existing diffusion language models and that it can be $100\times\sim600\times$ faster when achieving comparable results. Our code is available at https://github.com/microsoft/ProphetNet/tree/master/AR-diffusion.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑