arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2001.11314cs.CLcs.LG
收藏

ERNIE-GEN:一种用于自然语言生成的增强型多流预训练和微调框架

ERNIE-GEN: An Enhanced Multi-Flow Pre-training and Fine-tuning Framework for Natural Language Generation

Baidu Inc.(百度公司)

Dongling Xiao, Han Zhang, Yukun Li, Yu Sun, Hao Tian, Hua Wu, Haifeng Wang

摘要:

现有的自然语言生成的预训练工作很少关注下游任务的暴露偏差问题。为了解决这个问题,我们提出了一个增强的多流序列到序列预训练和微调框架ERNIE-GEN,它通过填充生成机制和噪声感知生成方法弥合了训练和推理之间的差异。为了使生成更接近人类的写作模式,该框架引入了一个逐跨度生成流程,该流程训练模型连续预测语义完整的跨度,而不是逐词预测。与现有的预训练方法不同,ERNIE-GEN采用多粒度目标采样来构建预训练数据,增强了编码器和解码器之间的相关性。实验结果表明,ERNIE-GEN在一系列语言生成任务上,包括抽象摘要(Gigaword和CNN/DailyMail)、问题生成(SQuAD)、对话生成(Persona-Chat)和生成式问答(CoQA),用更少量的预训练数据和参数实现了最先进的结果。

英文摘要:

Current pre-training works in natural language generation pay little attention to the problem of exposure bias on downstream tasks. To address this issue, we propose an enhanced multi-flow sequence to sequence pre-training and fine-tuning framework named ERNIE-GEN, which bridges the discrepancy between training and inference with an infilling generation mechanism and a noise-aware generation method. To make generation closer to human writing patterns, this framework introduces a span-by-span generation flow that trains the model to predict semantically-complete spans consecutively rather than predicting word by word. Unlike existing pre-training methods, ERNIE-GEN incorporates multi-granularity target sampling to construct pre-training data, which enhances the correlation between encoder and decoder. Experimental results demonstrate that ERNIE-GEN achieves state-of-the-art results with a much smaller amount of pre-training data and parameters on a range of language generation tasks, including abstractive summarization (Gigaword and CNN/DailyMail), question generation (SQuAD), dialogue generation (Persona-Chat) and generative question answering (CoQA).

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑