arXiv 2511.028332025-12-20cs.LGcs.CL

在良好的GRACE中：为知识蒸馏选择原则性教师

In Good GRACEs: Principled Teacher Selection for Knowledge Distillation

Princeton Language and Intelligence（普林斯顿语言与智能研究所）；Kempner Institute, Harvard（哈佛凯普纳研究所）；Microsoft Research, New York（微软研究院（纽约））；University of Pennsylvania（宾夕法尼亚大学）

Abhishek Panigrahi, Bingbin Liu, Sadhika Malladi, Sham Kakade, Surbhi Goel

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

GRACE通过量化教师有效性，为知识蒸馏提供原则性教师选择和蒸馏指导，提升学生模型性能。

摘要：

知识蒸馏是一种高效策略，利用由大型“教师”语言模型生成的数据来训练较小的“学生”模型，但为特定的学生-任务组合选择最佳教师需要昂贵的试错过程。我们提出了一种轻量级的评分称为GRACE，用于量化教师对后训练学生模型的有效性。GRACE测量学生梯度的分布属性，而无需访问验证者、教师日志、教师内部信息或测试数据。从信息论的角度来看，GRACE与梯度算法的留一法稳定性相关，这控制了蒸馏学生的一般化性能。在GSM8K和MATH上，GRACE与蒸馏LLaMA和OLMo学生的表现相关性很强（相关性高达86%）。特别是，使用GRACE选择的教师训练学生，可以比简单地使用表现最好的教师提高性能多达7.4%。此外，GRACE可以指导蒸馏中的关键设计选择，包括（1）在从教师生成时使用最佳温度，（2）在给定大小限制下使用最佳教师，以及（3）在特定模型家族中使用最佳教师。总体而言，我们的发现表明，GRACE能够高效且有效地识别给定学生强兼容的教师，并提供细粒度的蒸馏指导。

英文摘要：

Knowledge distillation is an efficient strategy to use data generated by large "teacher" language models to train smaller capable "student" models, but selecting the optimal teacher for a specific student-task combination requires expensive trial-and-error. We propose a lightweight score called GRACE to quantify how effective a teacher will be for post-training a student model. GRACE measures distributional properties of the student's gradients without access to a verifier, teacher logits, teacher internals, or test data. From an information-theoretic perspective, GRACE connects to leave-one-out stability of gradient-based algorithms, which controls the generalization performance of the distilled students. On GSM8K and MATH, GRACE correlates strongly (up to 86% Spearman correlation) with the performance of the distilled LLaMA and OLMo students. In particular, training a student using the GRACE-selected teacher can improve the performance by up to 7.4% over naively using the best-performing teacher. Further, GRACE can provide guidance on crucial design choices in distillation, including (1) the best temperature to use when generating from the teacher, (2) the best teacher to use given a size constraint, and (3) the best teacher to use within a specific model family. Altogether, our findings demonstrate that GRACE can efficiently and effectively identify a strongly compatible teacher for a given student and provide fine-grained guidance on how to perform distillation.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。