arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2004.10454cs.CLcs.LGcs.SDeess.AS
收藏

序列生成的非自回归模型研究

A Study of Non-autoregressive Model for Sequence Generation

Zhejiang University(浙江大学);Microsoft Research Asia(微软亚洲研究院);Microsoft STC Asia(微软亚洲互联网工程院)

Yi Ren, Jinglin Liu, Xu Tan, Zhou Zhao, Sheng Zhao, Tie-Yan Liu

摘要:

非自回归(NAR)模型并行生成序列的所有标记,与自回归(AR)模型相比,生成速度更快,但精度较低。已经提出了包括知识蒸馏和源-目标对准的不同技术,以在诸如神经机器翻译(NMT)、自动语音识别(ASR)和文本到语音(TTS)的各种任务中弥合AR和NAR模型之间的差距。在这些技术的帮助下,NAR模型在某些任务中可以赶上AR模型的准确性,但在另一些任务中则不然。在这项工作中,我们进行了研究,以了解NAR序列生成的困难,并试图回答:(1)为什么NAR模型可以赶上AR模型在某些任务,但不是所有?(2)为什么像知识蒸馏和源目标对齐这样的技术可以帮助NAR模型。由于AR和NAR模型之间的主要区别在于NAR模型不使用目标标记之间的依赖性,而AR模型使用,因此直观地,NAR序列生成的难度在很大程度上取决于目标标记之间的依赖性的强度。为了量化这种依赖关系,我们提出了一个分析模型,称为CoMMA来描述不同的NAR序列生成任务的难度。我们有几个有趣的发现:1)在NMT,ASR和TTS任务中,ASR具有最大的目标-令牌依赖性,而TTS具有最小的目标-令牌依赖性。2)知识蒸馏降低了目标序列中的目标-标记依赖性,从而提高了NAR模型的精度。3)源-目标对齐约束鼓励目标标记对源标记的依赖性,从而简化了NAR模型的训练。

英文摘要:

Non-autoregressive (NAR) models generate all the tokens of a sequence in parallel, resulting in faster generation speed compared to their autoregressive (AR) counterparts but at the cost of lower accuracy. Different techniques including knowledge distillation and source-target alignment have been proposed to bridge the gap between AR and NAR models in various tasks such as neural machine translation (NMT), automatic speech recognition (ASR), and text to speech (TTS). With the help of those techniques, NAR models can catch up with the accuracy of AR models in some tasks but not in some others. In this work, we conduct a study to understand the difficulty of NAR sequence generation and try to answer: (1) Why NAR models can catch up with AR models in some tasks but not all? (2) Why techniques like knowledge distillation and source-target alignment can help NAR models. Since the main difference between AR and NAR models is that NAR models do not use dependency among target tokens while AR models do, intuitively the difficulty of NAR sequence generation heavily depends on the strongness of dependency among target tokens. To quantify such dependency, we propose an analysis model called CoMMA to characterize the difficulty of different NAR sequence generation tasks. We have several interesting findings: 1) Among the NMT, ASR and TTS tasks, ASR has the most target-token dependency while TTS has the least. 2) Knowledge distillation reduces the target-token dependency in target sequence and thus improves the accuracy of NAR models. 3) Source-target alignment constraint encourages dependency of a target token on source tokens and thus eases the training of NAR models.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑