BD-KD：平衡在线知识蒸馏中的散度
BD-KD: Balancing the Divergences for Online Knowledge Distillation
AI总结：
BD-KD通过平衡前向与反向KL散度的样本级加权，在在线知识蒸馏中同时提升紧凑模型的精度与校准，无需事后重校准，实验验证其优于现有方法。
摘要：
我们解决了为边缘设备生成可信且精确的紧凑模型的挑战。虽然知识蒸馏（KD）在实现高精度性能方面改善了模型压缩，但这些紧凑模型的校准却被忽视了。我们引入了BD-KD（平衡散度知识蒸馏），一种基于logit的在线KD框架。BD-KD同时提升精度和模型校准，无需事后重校准技术，这些技术会给整体训练流程增加计算开销并降低性能。我们的方法通过调整学生和教师损失上的传统在线蒸馏损失，采用前向和反向Kullback-Leibler散度的样本级加权，鼓励以学生为中心的训练。该策略平衡了学生网络的置信度并提升了性能。在CIFAR10、CIFAR100、TinyImageNet和ImageNet数据集以及多种架构上的实验表明，与最近的在线KD方法相比，校准和精度均有改善。
英文摘要：
We address the challenge of producing trustworthy and accurate compact models for edge devices. While Knowledge Distillation (KD) has improved model compression in terms of achieving high accuracy performance, calibration of these compact models has been overlooked. We introduce BD-KD (Balanced Divergence Knowledge Distillation), a framework for logit-based online KD. BD-KD enhances both accuracy and model calibration simultaneously, eliminating the need for post-hoc recalibration techniques, which add computational overhead to the overall training pipeline and degrade performance. Our method encourages student-centered training by adjusting the conventional online distillation loss on both the student and teacher losses, employing sample-wise weighting of forward and reverse Kullback-Leibler divergence. This strategy balances student network confidence and boosts performance. Experiments across CIFAR10, CIFAR100, TinyImageNet, and ImageNet datasets, and various architectures demonstrate improved calibration and accuracy compared to recent online KD methods.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
