arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 1903.09722cs.CL
收藏

用于语言生成的预训练语言模型表示

Pre-trained Language Model Representations for Language Generation

Facebook AI Research(Facebook人工智能研究(Facebook AI Research));Menlo Park, CA(加利福尼亚州门洛帕克(Menlo Park, CA))

Sergey Edunov, Alexei Baevski, Michael Auli

摘要:

预先训练的语言模型表示在广泛的语言理解任务中已经取得了成功。在本文中,我们研究了将预训练的表示集成到序列到序列模型中的不同策略,并将其应用于神经机器翻译和抽象摘要。我们发现,预先训练的表示在添加到编码器网络时是最有效的,这只会使推理速度降低14%。我们在机器翻译中的实验显示,在模拟的资源贫乏设置中,增益高达5.3 BLEU。虽然回报随着更多的标记数据而减少,但当数百万个标记对可用时,我们仍然观察到改进。最后,在摘要方面,我们在CNN/DailyMail的全文版上取得了新的进展。

英文摘要:

Pre-trained language model representations have been successful in a wide range of language understanding tasks. In this paper, we examine different strategies to integrate pre-trained representations into sequence to sequence models and apply it to neural machine translation and abstractive summarization. We find that pre-trained representations are most effective when added to the encoder network which slows inference by only 14%. Our experiments in machine translation show gains of up to 5.3 BLEU in a simulated resource-poor setup. While returns diminish with more labeled data, we still observe improvements when millions of sentence-pairs are available. Finally, on abstractive summarization we achieve a new state of the art on the full text version of CNN/DailyMail.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑