UT5：通过展开去噪预训练非自回归T5
UT5: Pretraining Non autoregressive T5 with unrolled denoising
AI总结：
针对自回归模型顺序解码瓶颈，提出UT5，通过展开去噪预训练非自回归T5，在SQuAD问题生成和XSum等下游任务取得最先进结果。
摘要：
近年来，基于Transformer的大型语言模型在自然语言生成方面取得了巨大进展。然而，要解码K个令牌，自回归模型需要进行K次顺序前向传递，这可能成为大型语言模型的性能瓶颈。许多非自回归（NAR）研究旨在解决这一顺序性瓶颈，尽管许多研究专注于监督基准测试中的专用架构。在这项工作中，我们通过展开去噪研究了非自回归T5模型的无监督预训练，并在下游生成任务（如SQuAD问题生成和XSum）中展示了其最先进（SoTA）的结果。
英文摘要：
Recent advances in Transformer-based Large Language Models have made great strides in natural language generation. However, to decode K tokens, an autoregressive model needs K sequential forward passes, which may be a performance bottleneck for large language models. Many non-autoregressive (NAR) research are aiming to address this sequentiality bottleneck, albeit many have focused on a dedicated architecture in supervised benchmarks. In this work, we studied unsupervised pretraining for non auto-regressive T5 models via unrolled denoising and shown its SoTA results in downstream generation tasks such as SQuAD question generation and XSum.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
