arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2401.12997cs.CL
收藏

基于掩码生成特征方法的渐进式蒸馏用于知识图谱补全

Progressive Distillation Based on Masked Generation Feature Method for Knowledge Graph Completion

Anhui Province Key Laboratory of Multimodal Cognitive Computation, School of Computer Science and Technology, Anhui University(安徽大学计算机科学与技术学院多模态认知计算安徽省重点实验室);Department of Automation, Tsinghua University(清华大学自动化系)

Cunhang Fan, Yujie Chen, Jun Xue, Yonghui Kong, Jianhua Tao, Zhao Lv

AI总结:

本文针对知识图谱补全任务提出基于掩码生成特征的渐进式蒸馏方法,通过预蒸馏教师模型和多等级学生模型实现高效知识迁移,在保持性能的同时显著降低模型参数。

摘要:

近年来,基于预训练语言模型(PLM)的知识图谱补全(KGC)模型展现出了良好的效果。然而,PLM 模型参数规模大、计算成本高,给其在下游任务中的应用带来了挑战。本文针对 KGC 任务提出了一种基于掩码生成特征的渐进式蒸馏方法,旨在显著降低预训练模型的复杂度。具体而言,我们首先对 PLM 进行预蒸馏以获得高质量的教师模型,并压缩 PLM 网络以获得多等级的学生模型。然而,传统的特征蒸馏存在教师模型中信息表示单一的局限性。为了解决这一问题,我们提出了教师-学生特征的掩码生成方法,使其包含更丰富的表示信息。此外,教师与学生之间的表示能力存在显著差距。因此,我们设计了一种渐进式蒸馏方法,对每个等级的学生模型进行蒸馏,从而实现从教师到学生的高效知识迁移。实验结果表明,预蒸馏阶段的模型超越了现有的最先进方法。此外,在渐进式蒸馏阶段,模型在保持一定性能水平的同时显著减少了模型参数。具体而言,较低等级学生模型的参数相比基线减少了 56.7%。

英文摘要:

In recent years, knowledge graph completion (KGC) models based on pre-trained language model (PLM) have shown promising results. However, the large number of parameters and high computational cost of PLM models pose challenges for their application in downstream tasks. This paper proposes a progressive distillation method based on masked generation features for KGC task, aiming to significantly reduce the complexity of pre-trained models. Specifically, we perform pre-distillation on PLM to obtain high-quality teacher models, and compress the PLM network to obtain multi-grade student models. However, traditional feature distillation suffers from the limitation of having a single representation of information in teacher models. To solve this problem, we propose masked generation of teacher-student features, which contain richer representation information. Furthermore, there is a significant gap in representation ability between teacher and student. Therefore, we design a progressive distillation method to distill student models at each grade level, enabling efficient knowledge transfer from teachers to students. The experimental results demonstrate that the model in the pre-distillation stage surpasses the existing state-of-the-art methods. Furthermore, in the progressive distillation stage, the model significantly reduces the model parameters while maintaining a certain level of performance. Specifically, the model parameters of the lower-grade student model are reduced by 56.7\% compared to the baseline.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑