arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2101.00416cs.CL
收藏

通过序列跨度重写改进序列到序列预训练

Improving Sequence-to-Sequence Pre-training via Sequence Span Rewriting

Stanford University(斯坦福大学);Microsoft Research Asia(微软亚洲研究院);University of California, San Diego(加利福尼亚大学圣迭戈分校);Beihang University(北京航空航天大学)

Wangchunshu Zhou, Tao Ge, Canwen Xu, Ke Xu, Furu Wei

摘要:

在本文中,我们概括了文本填充(例如,掩蔽的语言模型),提出序列跨度重写(SSR)作为自监督序列到序列(seq2seq)的预训练目标。SSR通过监督模型将不完美的跨度重写为基本事实,为文本表示提供了更细粒度的学习信号,并且它比文本填充更一致,其中许多下游seq2seq任务将源句子重写为目标句子。我们在各种seq2seq任务上使用T5模型进行的实验表明,SSR可以显著改善seq2seq预训练。此外,我们观察到SSR特别有助于改进具有强大的不完美跨度生成器的小规模seq2seq模型的预训练,这表明将知识从大模型转移到较小的seq2seq预训练模型的新视角。

英文摘要:

In this paper, we generalize text infilling (e.g., masked language models) by proposing Sequence Span Rewriting (SSR) as a self-supervised sequence-to-sequence (seq2seq) pre-training objective. SSR provides more fine-grained learning signals for text representations by supervising the model to rewrite imperfect spans to ground truth, and it is more consistent than text infilling with many downstream seq2seq tasks that rewrite a source sentences into a target sentence. Our experiments with T5 models on various seq2seq tasks show that SSR can substantially improve seq2seq pre-training. Moreover, we observe SSR is especially helpful to improve pre-training a small-size seq2seq model with a powerful imperfect span generator, which indicates a new perspective of transferring knowledge from a large model to a smaller model for seq2seq pre-training.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑