基于条件互信息的知识蒸馏贝叶斯条件分布估计
Bayes Conditional Distribution Estimation for Knowledge Distillation Based on Conditional Mutual Information
AI总结：
针对知识蒸馏中传统最大对数似然方法估计贝叶斯条件分布精度不足的问题，提出最大条件互信息估计方法训练教师模型，提升学生分类准确率，在少样本、零样本场景下增益更显著。
摘要：
学界普遍认为，在知识蒸馏（KD）中，教师模型的作用是对未知的贝叶斯条件概率分布（BCPD）进行估计，以供学生模型训练过程使用。传统方法中，该估计值通过最大对数似然（MLL）方法训练教师模型得到。为了改进知识蒸馏中的这一估计效果，本文将条件互信息（CMI）概念引入贝叶斯条件概率分布的估计过程，提出了一种名为最大条件互信息（MCMI）的新型估计方法。具体而言，在MCMI估计中，训练教师模型时会同时最大化教师模型的对数似然值与条件互信息。通过Eigen-CAM（特征类激活映射）进一步验证，最大化教师模型的CMI值可使教师模型捕捉图像簇中更多的上下文信息。通过开展全面的实验我们发现，在各类主流先进的知识蒸馏框架中，采用经MCMI估计训练的教师模型替代经MLL估计训练的教师模型，学生模型的分类准确率均得到持续提升，最高增幅达3.32%。这表明MCMI方法提供的教师模型BCPD估计比MLL方法的估计结果更准确。此外，我们还发现这种学生模型准确率的提升在零样本和少样本场景下更为显著。值得注意的是，当学生模型仅能使用5%的训练样本时（少样本场景），学生模型的准确率最高提升5.72%；而对于未见过的类别（零样本场景），学生模型的准确率从0%提升至最高84%。相关代码已开源在https://github.com/iclr2024mcmi/ICLRMCMI。
英文摘要：
It is believed that in knowledge distillation (KD), the role of the teacher is to provide an estimate for the unknown Bayes conditional probability distribution (BCPD) to be used in the student training process. Conventionally, this estimate is obtained by training the teacher using maximum log-likelihood (MLL) method. To improve this estimate for KD, in this paper we introduce the concept of conditional mutual information (CMI) into the estimation of BCPD and propose a novel estimator called the maximum CMI (MCMI) method. Specifically, in MCMI estimation, both the log-likelihood and CMI of the teacher are simultaneously maximized when the teacher is trained. Through Eigen-CAM, it is further shown that maximizing the teacher's CMI value allows the teacher to capture more contextual information in an image cluster. Via conducting a thorough set of experiments, we show that by employing a teacher trained via MCMI estimation rather than one trained via MLL estimation in various state-of-the-art KD frameworks, the student's classification accuracy consistently increases, with the gain of up to 3.32\%. This suggests that the teacher's BCPD estimate provided by MCMI method is more accurate than that provided by MLL method. In addition, we show that such improvements in the student's accuracy are more drastic in zero-shot and few-shot settings. Notably, the student's accuracy increases with the gain of up to 5.72\% when 5\% of the training samples are available to the student (few-shot), and increases from 0\% to as high as 84\% for an omitted class (zero-shot). The code is available at \url{https://github.com/iclr2024mcmi/ICLRMCMI}.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
