arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2409.12586cs.CL
收藏

高效知识蒸馏:利用教师模型洞察增强小型语言模型

Efficient Knowledge Distillation: Empowering Small Language Models with Teacher Model Insights

Osnabrueck University(奥斯纳布吕克大学);Institute of Cognitive Science(认知科学研究所)

Mohamad Ballout, Ulf Krumnack, Gunther Heidemann, Kai-Uwe Kühnberger

AI总结:

提出一种高效知识蒸馏方法,利用约30亿参数的教师模型提取关键标记作为理由,指导学生模型提升性能,在四个数据集上优于标准微调和现有蒸馏方法,并发现68%情况下提取标记为真实答案。

摘要:

增强小型语言模型以应用于实际部署,是研究界面临的重大挑战。由于使用大型语言模型的困难和成本,研究人员正在寻求有效部署任务特定小型模型的方法。在这项工作中,我们提出了一种简单而有效的知识蒸馏方法,以提升小型语言模型的性能。我们的方法利用一个约30亿参数的教师模型,识别其决策过程中最具影响力的标记。这些标记基于它们相对于输出的归因分数从输入中提取,使用诸如显著性图等方法。然后,这些重要标记作为理由提供给一个学生模型,旨在蒸馏教师模型的知识。该方法已被证明有效,通过在四个多样化数据集上的测试,显示出相对于标准微调方法和最先进的知识蒸馏模型的改进。此外,我们通过分析从教师模型提取的重要标记,探讨了模型成功的解释。我们的发现表明,在68%的情况下,特别是在标签是答案一部分的数据集(如多项选择题)中,提取的标记是真实答案的一部分。

英文摘要:

Enhancing small language models for real-life application deployment is a significant challenge facing the research community. Due to the difficulties and costs of using large language models, researchers are seeking ways to effectively deploy task-specific small models. In this work, we introduce a simple yet effective knowledge distillation method to improve the performance of small language models. Our approach utilizes a teacher model with approximately 3 billion parameters to identify the most influential tokens in its decision-making process. These tokens are extracted from the input based on their attribution scores relative to the output, using methods like saliency maps. These important tokens are then provided as rationales to a student model, aiming to distill the knowledge of the teacher model. This method has proven to be effective, as demonstrated by testing it on four diverse datasets, where it shows improvement over both standard fine-tuning methods and state-of-the-art knowledge distillation models. Furthermore, we explore explanations of the success of the model by analyzing the important tokens extracted from the teacher model. Our findings reveal that in 68\% of cases, specifically in datasets where labels are part of the answer, such as multiple-choice questions, the extracted tokens are part of the ground truth.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑