arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2210.00907cs.CL
收藏

面向事实知识注入的掩蔽语言建模和适配器的有效性

The Effectiveness of Masked Language Modeling and Adapters for Factual Knowledge Injection

University of Oslo(奥斯陆大学)

Sondre Wold

摘要:

本文研究了将事实知识注入到大型预训练语言模型中的问题。我们训练适配器模块上的部分ConceptNet知识图谱使用掩蔽语言建模目标和评估的成功的方法进行了一系列的探测实验上的LAMA探针。不同配置的平均P@K曲线表明,该技术是有效的,增加性能的LAMA探针的子集为大值的k,通过添加少至2.1%的额外参数,以原始模型。

英文摘要:

This paper studies the problem of injecting factual knowledge into large pre-trained language models. We train adapter modules on parts of the ConceptNet knowledge graph using the masked language modeling objective and evaluate the success of the method by a series of probing experiments on the LAMA probe. Mean P@K curves for different configurations indicate that the technique is effective, increasing the performance on subsets of the LAMA probe for large values of k by adding as little as 2.1% additional parameters to the original models.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑