温度自适应变换教师匹配
Temperature-Adaptive Transformed Teacher Matching
浏览论文内容
中文总结 AI 辅助
该研究针对变换教师匹配（TTM）引入逐样本逆温度更新，提出温度自适应方法，在图像分类蒸馏基准上可提升TTM和WTTM性能，优于或媲美现有温度自适应蒸馏基线。
中文摘要 AI 辅助
温度缩放是知识蒸馏的核心组件，但其作用与效果尚未被完全理解。变换教师匹配（TTM）通过仅将温度缩放应用于教师分布，并将所得目标解释为对学生施加隐式Rényi熵正则化的标准蒸馏，阐明了温度缩放的作用。然而，TTM仍依赖固定温度，未指定教师侧温度如何针对单个样本进行自适应调整。本文针对TTM引入逐样本逆温度更新，方法是局部最小化经温度缩放的教师分布与学生预测之间的Kullback-Leibler散度。我们推导了关于逆温度的闭式一阶和二阶导数，证明其可通过变换教师加权下中心化教师与学生logit的方差和协方差统计量表示，由此得到高效的曲率感知更新，仅需一次softmax计算和固定数量的类别加权求和。在标准图像分类蒸馏基准上的实验表明，我们的温度自适应方法通常可提升TTM和WTTM的性能，同时与现有温度自适应蒸馏基线相比具有竞争力或更优。
英文摘要
Temperature scaling is a core component of knowledge distillation, yet its role and effect are still not fully understood. Transformed Teacher Matching (TTM) clarifies the role of temperature scaling by applying it only to the teacher distribution and interpreting the resulting objective as standard distillation with an implicit Rényi entropy regularization on the student. However, TTM still relies on a fixed temperature and does not specify how the teacher-side temperature should be adapted for individual samples. In this paper, we introduce a sample-wise inverse-temperature update for TTM by locally minimizing the Kullback-Leibler divergence between the temperature-scaled teacher distribution and the student's prediction. We derive closed-form first and second derivatives with respect to the inverse temperature, and show that they can be expressed using variance and covariance statistics of centered teacher and student logits under the transformed teacher weighting. This yields an efficient curvature-aware update that requires one softmax evaluation and a constant number of class-wise weighted sums. Experiments on standard image classification distillation benchmarks show that our temperature adaptation generally improves TTM and WTTM, while remaining competitive with or outperforming prior temperature-adaptive distillation baselines.
发表机构
- Graduate School of Advanced Science and Engineering Hiroshima University（广岛大学先进科学与工程学院）
- Intelligent Production Technology Research & Development Center for Aerospace Gifu University（岐阜大学航空航天智能生产技术研究开发中心）
机构由 AI 辅助整理，请以论文原文为准。
