教师模型误设定下的知识蒸馏：教师模仿与任务性能之间差距的序参量分析
Knowledge Distillation under Teacher Misspecification: An Order-Parameter Analysis of the Gap between Teacher Mimicry and Task Performance
- Nihon University（日本大学）
- The Institute of Statistical Mathematics（统计数理研究所）
机构由 AI 辅助整理，请以论文原文为准。
AI总结：
该研究在极简三方模型中分析教师模型误设定下的知识蒸馏，证明蒸馏误差对失配强度不变但真实误差及对应差距随其递增，警示仅用教师模仿指标评估蒸馏的局限性。
AI中文摘要：
知识蒸馏旨在训练小型学生模型以复现大型教师模型的输出，其进展通常通过师生差异来监测。然而，最终关注的量是学生相对于真实任务的误差。我们在一个极简三方模型中研究这两个目标之间的关系：真实教师（生成模型）、教师和学生均为软委员会机，其中真实教师包含教师无法表示的共享潜在因子，失配强度由单一标量$\boldsymbol{\rmiss}$控制。在在线蒸馏的序参量描述中，并利用误差函数激活下所有误差的闭式（反正弦型）表达式，我们证明学习动态和蒸馏误差$\boldsymbol{\rm E_{ts}}$对$\boldsymbol{\rmiss}$完全不变，而真实误差$\boldsymbol{\rm E_{tzs}}$和差距$\boldsymbol{\rm \triangle = E_{tzs} - E_{ts}}$随$\boldsymbol{\rmiss}$严格递增，其速率被真实教师的复杂度$\boldsymbol{M_0}$线性放大。真实教师复杂度与学生容量构成平面上的数值相图证实了预测的变形：$\boldsymbol{\rm E_{ts}}$的等高线未移动，而$\boldsymbol{\rm E_{tzs}}$的分布系统性上升，且教师误设定 regime（区域）中模仿成功但任务失败的情况随$\boldsymbol{\rmiss}$扩大。结果对仅通过教师模仿指标评估蒸馏给出了定量警示，并将差距$\boldsymbol{\rm \triangle}$确定为区分教师误设定与容量受限失败的最小诊断指标。
英文摘要：
Knowledge distillation trains a small student model to reproduce the outputs of a large teacher model, and its progress is typically monitored through the teacher--student discrepancy. The quantity of ultimate interest, however, is the student's error with respect to the true task. We study the relation between these two objectives in a minimal three-party model, a true teacher (generative model), a teacher, and a student, all soft committee machines, in which the true teacher contains a shared latent factor that the teacher cannot represent, with mismatch strength controlled by a single scalar $\dmiss$. Within an order-parameter description of online distillation, and exploiting closed-form (arcsine-type) expressions for all errors under error-function activations, we prove that the learning dynamics and the distillation error $\Ets$ are exactly invariant to $\dmiss$, whereas the true error $\Etzs$ and the gap $Δ=\Etzs-\Ets$ are strictly increasing in $\dmiss$, with a rate that is amplified linearly by the complexity $M_0$ of the true teacher. Numerical phase diagrams over the plane spanned by true-teacher complexity and student capacity confirm the predicted deformation: the contours of $\Ets$ do not move while the landscape of $\Etzs$ rises systematically, and a teacher-miss regime, where mimicry succeeds but the task fails, expands with $\dmiss$. The results give a quantitative warning against evaluating distillation solely through teacher-mimicry metrics and identify the gap $Δ$ as a minimal diagnostic for distinguishing teacher-miss from capacity-limited failure.
