arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2403.11203cs.CL
收藏

TRELM:面向知识增强语言模型的鲁棒高效预训练

TRELM: Towards Robust and Efficient Pre-training for Knowledge-Enhanced Language Models

Alibaba Group(阿里巴巴集团)

Junbing Yan, Chengyu Wang, Taolin Zhang, Xiaofeng He, Jun Huang, Longtao Huang, Hui Xue, Wei Zhang

AI总结:

针对知识增强语言模型预训练中实体表示优化不足和参数更新开销大的问题,提出TRELM框架,通过鲁棒知识注入、记忆库和动态知识路由选择性更新FFN参数,实现至少50%的预训练加速并提升下游任务性能。

摘要:

知识增强预训练语言模型(KEPLMs)是利用外部知识来增强语言理解的预训练模型。以往的语言模型通过融入从知识图谱中的关系三元组学习到的与知识相关的预训练任务来促进知识获取。然而,这些模型并未优先学习实体相关令牌的嵌入。此外,更新KEPLMs中的全部参数集在计算上代价高昂。本文介绍了TRELM,一个面向知识增强语言模型的鲁棒且高效的预训练框架。我们观察到,文本语料中的实体通常遵循长尾分布,其中某些实体的表示优化不充分,从而阻碍了KEPLMs的预训练过程。为解决这一问题,我们采用一种鲁棒的方法来注入知识三元组,并利用知识增强的记忆库来捕获有价值的信息。此外,更新存储事实知识的前馈网络(FFNs)中的一小部分神经元既充分又高效。具体而言,我们利用动态知识路由来识别FFNs中的知识路径,并在预训练期间选择性地更新参数。实验结果表明,TRELM将预训练时间至少减少了50%,并在知识探测任务和多个知识感知的语言理解任务上优于其他KEPLMs。

英文摘要:

KEPLMs are pre-trained models that utilize external knowledge to enhance language understanding. Previous language models facilitated knowledge acquisition by incorporating knowledge-related pre-training tasks learned from relation triples in knowledge graphs. However, these models do not prioritize learning embeddings for entity-related tokens. Moreover, updating the entire set of parameters in KEPLMs is computationally demanding. This paper introduces TRELM, a Robust and Efficient Pre-training framework for Knowledge-Enhanced Language Models. We observe that entities in text corpora usually follow the long-tail distribution, where the representations of some entities are suboptimally optimized and hinder the pre-training process for KEPLMs. To tackle this, we employ a robust approach to inject knowledge triples and employ a knowledge-augmented memory bank to capture valuable information. Furthermore, updating a small subset of neurons in the feed-forward networks (FFNs) that store factual knowledge is both sufficient and efficient. Specifically, we utilize dynamic knowledge routing to identify knowledge paths in FFNs and selectively update parameters during pre-training. Experimental results show that TRELM reduces pre-training time by at least 50% and outperforms other KEPLMs in knowledge probing tasks and multiple knowledge-aware language understanding tasks.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑