BANG:通过大规模预训练桥接自回归和非自回归生成
BANG: Bridging Autoregressive and Non-autoregressive Generation with Large Scale Pretraining
摘要:
在本文中,我们提出了BANG,一个新的预训练模型,以弥合自回归(AR)和非自回归(NAR)生成之间的差距。AR和NAR的生成可以统一地看作是在多大程度上可以关注先前的token,并且BANG通过设计用于大规模预训练的新型模型结构来桥接AR和NAR生成。预训练的BANG模型可以同时支持AR、NAR和半NAR生成,以满足不同的需求。在问题生成(SQuAD 1.1)、摘要(XSum)和对话生成(PersonaChat)上的实验表明,BANG显著提高了NAR和半NAR性能,并达到了与强AR预训练模型相当的性能。与半NAR强基线相比,BANG在SQuAD 1.1和XSum的总评分中分别实现了14.01和5.24的绝对改善。此外,与强NAR基线相比,BANG在SQuAD、XSUM和PersonaChat的总得分分别实现了10.73、6.39和5.90的绝对改善。
英文摘要:
In this paper, we propose BANG, a new pretraining model to Bridge the gap between Autoregressive (AR) and Non-autoregressive (NAR) Generation. AR and NAR generation can be uniformly regarded as to what extent previous tokens can be attended, and BANG bridges AR and NAR generation by designing a novel model structure for large-scale pretraining. The pretrained BANG model can simultaneously support AR, NAR and semi-NAR generation to meet different requirements. Experiments on question generation (SQuAD 1.1), summarization (XSum) and dialogue generation (PersonaChat) show that BANG improves NAR and semi-NAR performance significantly as well as attaining comparable performance with strong AR pretrained models. Compared with the semi-NAR strong baselines, BANG achieves absolute improvements of 14.01 and 5.24 in the overall scores of SQuAD 1.1 and XSum, respectively. In addition, BANG achieves absolute improvements of 10.73, 6.39 and 5.90 in the overall scores of SQuAD, XSUM and PersonaChat respectively compared with the strong NAR baselines.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。