Progressive Distillation Based on Masked Generation Feature Method for Knowledge Graph Completion
Anhui Province Key Laboratory of Multimodal Cognitive Computation, School of Computer Science and Technology, Anhui University(安徽大学计算机科学与技术学院多模态认知计算安徽省重点实验室);Department of Automation, Tsinghua University(清华大学自动化系)
In recent years, knowledge graph completion (KGC) models based on pre-trained language model (PLM) have shown promising results. However, the large number of parameters and high computational cost of PLM models pose challenges for their application in downstream tasks. This paper proposes a progressive distillation method based on masked generation features for KGC task, aiming to significantly reduce the complexity of pre-trained models. Specifically, we perform pre-distillation on PLM to obtain high-quality teacher models, and compress the PLM network to obtain multi-grade student models. However, traditional feature distillation suffers from the limitation of having a single representation of information in teacher models. To solve this problem, we propose masked generation of teacher-student features, which contain richer representation information. Furthermore, there is a significant gap in representation ability between teacher and student. Therefore, we design a progressive distillation method to distill student models at each grade level, enabling efficient knowledge transfer from teachers to students. The experimental results demonstrate that the model in the pre-distillation stage surpasses the existing state-of-the-art methods. Furthermore, in the progressive distillation stage, the model significantly reduces the model parameters while maintaining a certain level of performance. Specifically, the model parameters of the lower-grade student model are reduced by 56.7\% compared to the baseline.
Improving Reasoning Capabilities in Small Models through Mixture-of-Layers Distillation with Stepwise Attention on Key Information
Institute of Information Engineering, Chinese Academy of Sciences(中国科学院信息工程研究所);School of Cyber Security, University of Chinese Academy of Sciences(中国科学院大学网络安全学院)
T$^\star$: Progressive Block Scaling for Masked Diffusion Language Models Through Trajectory Aware Reinforcement Learning
Shanghai Academy of AI for Science(上海人工智能科学研究院);Shanghai Innovation Institute(上海创新研究院);Fudan University(复旦大学);School of Mathematical Sciences(数学科学学院);Shanghai Jiao Tong University(上海交通大学);Carnegie Mellon University(卡内基梅隆大学)
Sparse Progressive Distillation: Resolving Overfitting under Pretrain-and-Finetune Paradigm
University of Connecticut(康涅狄格大学);Penn State University(宾夕法尼亚州立大学);Moffett AI(莫菲特人工智能公司);Northeastern University(东北大学);Stevens Institute of Technology(史蒂文斯理工学院);University of Texas at San Antonio(德克萨斯大学圣安东尼奥分校)