MLKD-BERT：面向预训练语言模型的多层级知识蒸馏
MLKD-BERT: Multi-level Knowledge Distillation for Pre-trained Language Models
AI总结：
本文提出MLKD-BERT，通过在教师-学生框架中蒸馏多层级知识并灵活设置学生注意力头数量，在GLUE和抽取式问答任务上超越现有知识蒸馏方法，同时显著降低推理时间。
摘要：
知识蒸馏是预训练语言模型压缩的有效技术。尽管现有知识蒸馏方法在最典型的模型BERT上表现良好，但仍可在两个方面进一步改进：关系级知识可被进一步挖掘以提升模型性能；学生注意力头数量的设置可以更灵活，以减少推理时间。因此，我们提出了一种新颖的知识蒸馏方法MLKD-BERT，在教师-学生框架中蒸馏多层级知识。在GLUE基准和抽取式问答任务上的大量实验表明，我们的方法在BERT上优于最先进的知识蒸馏方法。此外，MLKD-BERT可以灵活设置学生注意力头数量，从而在性能下降很小的情况下大幅减少推理时间。
英文摘要：
Knowledge distillation is an effective technique for pre-trained language model compression. Although existing knowledge distillation methods perform well for the most typical model BERT, they could be further improved in two aspects: the relation-level knowledge could be further explored to improve model performance; and the setting of student attention head number could be more flexible to decrease inference time. Therefore, we are motivated to propose a novel knowledge distillation method MLKD-BERT to distill multi-level knowledge in teacher-student framework. Extensive experiments on GLUE benchmark and extractive question answering tasks demonstrate that our method outperforms state-of-the-art knowledge distillation methods on BERT. In addition, MLKD-BERT can flexibly set student attention head number, allowing for substantial inference time decrease with little performance drop.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
