arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2107.02137cs.CL
收藏

ERNIE 3.0:大规模知识增强的语言理解和生成预训练

ERNIE 3.0: Large-scale Knowledge Enhanced Pre-training for Language Understanding and Generation

Baidu Inc.(百度公司)
Yu Sun, Shuohuan Wang, Shikun Feng, Siyu Ding, Chao Pang, Junyuan Shang, Jiaxiang Liu, Xuyi Chen, Yanbin Zhao, Yuxiang Lu, Weixin Liu, Zhihua Wu, Weibao Gong, J… 展开作者

Yu Sun, Shuohuan Wang, Shikun Feng, Siyu Ding, Chao Pang, Junyuan Shang, Jiaxiang Liu, Xuyi Chen, Yanbin Zhao, Yuxiang Lu, Weixin Liu, Zhihua Wu, Weibao Gong, Jianzhong Liang, Zhizhou Shang, Peng Sun, Wei Liu, Xuan Ouyang, Dianhai Yu, Hao Tian, Hua Wu, Haifeng Wang

摘要:

预训练模型在各种自然语言处理(NLP)任务中取得了最先进的结果。最近的工作,如T5和GPT-3已经表明,扩大预训练的语言模型可以提高其泛化能力。特别地,具有1750亿个参数的GPT-3模型显示了其强大的任务不可知zero-shot/少样本学习能力。尽管它们取得了成功,但这些大规模模型是在纯文本上训练的,没有引入语言知识和世界知识等知识。此外,大多数大规模模型都是以自回归的方式训练的。因此,这种传统的微调方法在解决下游语言理解任务时表现出相对较弱的性能。为了解决上述问题,我们提出了一个统一的框架ERNIE 3.0,用于预训练大规模知识增强模型。它融合了自回归网络和自动编码网络,使得训练好的模型可以很容易地通过少样本学习、零样本学习或微调来适应自然语言理解和生成任务。我们在由纯文本和大规模知识图谱组成的4 TB语料库上训练了具有100亿个参数的模型。实验结果表明,该模型在54个中文NLP任务上的性能优于最先进的模型,其英文版本在SuperGLUE基准测试(2021年7月3日)中获得第一名,超过人类性能+0.8%(90.6% vs. 89.8%)。

英文摘要:

Pre-trained models have achieved state-of-the-art results in various Natural Language Processing (NLP) tasks. Recent works such as T5 and GPT-3 have shown that scaling up pre-trained language models can improve their generalization abilities. Particularly, the GPT-3 model with 175 billion parameters shows its strong task-agnostic zero-shot/few-shot learning capabilities. Despite their success, these large-scale models are trained on plain texts without introducing knowledge such as linguistic knowledge and world knowledge. In addition, most large-scale models are trained in an auto-regressive way. As a result, this kind of traditional fine-tuning approach demonstrates relatively weak performance when solving downstream language understanding tasks. In order to solve the above problems, we propose a unified framework named ERNIE 3.0 for pre-training large-scale knowledge enhanced models. It fuses auto-regressive network and auto-encoding network, so that the trained model can be easily tailored for both natural language understanding and generation tasks with zero-shot learning, few-shot learning or fine-tuning. We trained the model with 10 billion parameters on a 4TB corpus consisting of plain texts and a large-scale knowledge graph. Empirical results show that the model outperforms the state-of-the-art models on 54 Chinese NLP tasks, and its English version achieves the first place on the SuperGLUE benchmark (July 3, 2021), surpassing the human performance by +0.8% (90.6% vs. 89.8%).

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑