跨学生设计的基于特征的知识蒸馏的对照研究
A Controlled Study of Feature-Based Knowledge Distillation Across Student Designs
浏览论文内容
中文总结 AI 辅助
该研究在CIFAR-100上对照分析不同特征知识蒸馏方法对不同学生模型的影响，发现FitNets弱于logit KD、注意力迁移效果因模型而异，固定辅助系数无法实现统一训练条件。
中文摘要 AI 辅助
知识蒸馏（Knowledge Distillation）训练更小的学生模型（student）以匹配更大的教师模型（teacher）的输出，基于特征的方法还会对齐中间表示，但这种额外约束可能对不同学生模型产生不同影响。我们在CIFAR-100数据集上开展研究，使用ResNet-50作为教师模型、宽度可控的CustomResNet系列以及MobileNetV2作为跨设计对比对象。对每个学生模型，我们将每种特征方法与匹配的logit-KD运行进行评估，二者使用相同的教师模型、优化器设置、训练计划和随机种子，且在多个种子上重复主要对比。结果显示，logit KD使所有测试学生模型均优于其随机初始化基线；注意力迁移（Attention Transfer）在CustomResNet系列内部与模型大小无明确关系，但其对该系列的平均效果为负，对MobileNetV2则为正；FitNets在全部15次配对运行中均弱于logit KD，在恒定深度的宽度扫描中，其差距随学生模型宽度增加而增大，不过不同深度的w=48学生模型未遵循该趋势；最后，相同的辅助系数在不同学生模型上产生不同的梯度尺度，表明固定系数无法创造统一的训练条件。
英文摘要
Knowledge distillation trains a smaller student to match the outputs of a larger teacher. Feature-based methods also align intermediate representations, but this extra constraint may affect students differently. We study this question on CIFAR-100 using a ResNet-50 teacher, a width-controlled CustomResNet family and MobileNetV2 as a cross-design comparison. For each student, we evaluate each feature method against a matched logit-KD run using the same teacher, optimizer settings, training schedule and seed. We repeat the main comparisons across multiple seeds. Logit KD improved every tested student over its scratch baseline. Attention Transfer showed no clear relationship with size inside the CustomResNet family, but its average effect was negative for that family and positive for MobileNetV2. FitNets was below logit KD in all 15 paired runs. Within the constant-depth width sweep, its gap increased for wider students, although the different-depth w=48 student did not follow this trend. Finally, the same auxiliary coefficient produced different gradient scales across students, showing that a fixed coefficient does not create a uniform training condition.
发表机构
- Georgia Institute of Technology（佐治亚理工学院）
机构由 AI 辅助整理，请以论文原文为准。
