教学启发式数据合成用于语言模型知识蒸馏
Pedagogically-Inspired Data Synthesis for Language Model Knowledge Distillation
AI总结：
本文提出基于教学原理的知识蒸馏框架，通过三阶段流程提升学生模型性能，在复杂推理任务中取得显著改进。
摘要：
从大型语言模型（LLMs）向小型模型的知识蒸馏已成为部署高效人工智能系统的关键技术。然而，当前通过合成数据进行蒸馏的方法缺乏教学意识，将知识转移视为一次性数据合成和训练任务，而非系统学习过程。在本文中，我们提出了一种新颖的教学启发式框架，用于LLM知识蒸馏，该框架借鉴了基本的教育原则。我们的方法引入了一个三阶段流程——知识标识符、组织者和适配器（IOA）——该流程系统地识别学生模型中的知识缺陷，通过渐进课程组织知识传递，并适应表示以匹配学生模型的认知能力。我们整合了布鲁姆的掌握学习原则和维果茨基的最近发展区理论，创造了一个动态的蒸馏过程，其中学生模型在掌握先决知识之前接近教师模型的性能，然后逐步引入新知识，以受控的、渐进的方式增加难度。使用LLaMA-3.1/3.2和Qwen2.5作为学生模型进行的广泛实验表明，IOA在基准蒸馏方法上实现了显著改进，学生模型在DollyEval上保留了教师性能的94.7%，同时参数使用少于1/10。我们的框架在复杂推理任务上表现尤为突出，在MATH任务上比最先进的基线提高了19.2%，在HumanEval上提高了22.3%。
英文摘要：
Knowledge distillation from Large Language Models (LLMs) to smaller models has emerged as a critical technique for deploying efficient AI systems. However, current methods for distillation via synthetic data lack pedagogical awareness, treating knowledge transfer as a one-off data synthesis and training task rather than a systematic learning process. In this paper, we propose a novel pedagogically-inspired framework for LLM knowledge distillation that draws from fundamental educational principles. Our approach introduces a three-stage pipeline -- Knowledge Identifier, Organizer, and Adapter (IOA) -- that systematically identifies knowledge deficiencies in student models, organizes knowledge delivery through progressive curricula, and adapts representations to match the cognitive capacity of student models. We integrate Bloom's Mastery Learning Principles and Vygotsky's Zone of Proximal Development to create a dynamic distillation process where student models approach teacher model's performance on prerequisite knowledge before advancing, and new knowledge is introduced with controlled, gradual difficulty increments. Extensive experiments using LLaMA-3.1/3.2 and Qwen2.5 as student models demonstrate that IOA achieves significant improvements over baseline distillation methods, with student models retaining 94.7% of teacher performance on DollyEval while using less than 1/10th of the parameters. Our framework particularly excels in complex reasoning tasks, showing 19.2% improvement on MATH and 22.3% on HumanEval compared with state-of-the-art baselines.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
