面向知识蒸馏的平衡散度
Balance Divergence for Knowledge Distillation
AI总结：
针对现有知识蒸馏用KL散度忽略教师暗知识负向部分、导致学生信息不平衡的问题，提出平衡散度蒸馏法，用反向KL散度补偿优化极小值建模，在多视觉任务上实现性能提升且易推广。
摘要：
知识蒸馏因能利用从复杂教师网络迁移的知识有效提升轻量学生网络的性能，已被广泛应用于计算机视觉任务处理中。现有大多数知识蒸馏方法采用Kullback-Leibler散度来模仿教师网络与学生网络之间的logit输出概率。然而，这类方法可能会忽略教师“暗知识”的负向部分，因为散度计算可能会忽视教师logit输出中微小概率带来的影响。这一缺陷可能导致蒸馏过程中logit模仿的性能次优，并造成学生网络获取的信息不平衡。本文研究了这种不平衡的影响，并提出了一种名为Balance Divergence Distillation（平衡散度蒸馏）的新方法。通过引入使用反向Kullback-Leibler散度的补偿操作，我们的方法能够改进对教师输出负向部分中极小值的建模，同时保留对正向部分的学习能力。此外，我们测试了不同温度系数调整的影响，该调整可进一步平衡知识迁移过程。我们在多个计算机视觉任务上评估了所提方法，包括图像分类和语义分割。评估结果显示，在CIFAR-100和ImageNet数据集上，我们的方法使轻量学生网络的准确率提升了1%~3%；在Cityscapes数据集上，PSP-ResNet18的mIoU提升了4.55%。实验表明，我们的方法是一种简单且高效的方案，可平滑应用于不同的知识蒸馏方法。
英文摘要：
Knowledge distillation has been widely adopted in computer vision task processing, since it can effectively enhance the performance of lightweight student networks by leveraging the knowledge transferred from cumbersome teacher networks. Most existing knowledge distillation methods utilize Kullback-Leibler divergence to mimic the logit output probabilities between the teacher network and the student network. Nonetheless, these methods may neglect the negative parts of the teacher's ''dark knowledge'' because the divergence calculations may ignore the effect of the minute probabilities from the teacher's logit output. This deficiency may lead to suboptimal performance in logit mimicry during the distillation process and result in an imbalance of information acquired by the student network. In this paper, we investigate the impact of this imbalance and propose a novel method, named Balance Divergence Distillation. By introducing a compensatory operation using reverse Kullback-Leibler divergence, our method can improve the modeling of the extremely small values in the negative from the teacher and preserve the learning capacity for the positive. Furthermore, we test the impact of different temperature coefficients adjustments, which may conducted to further balance for knowledge transferring. We evaluate the proposed method on several computer vision tasks, including image classification and semantic segmentation. The evaluation results show that our method achieves an accuracy improvement of 1%~3% for lightweight students on both CIFAR-100 and ImageNet dataset, and a 4.55% improvement in mIoU for PSP-ResNet18 on the Cityscapes dataset. The experiments show that our method is a simple yet highly effective solution that can be smoothly applied to different knowledge distillation methods.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
