利用能量函数最大化知识蒸馏的判别能力
Maximizing Discrimination Capability of Knowledge Distillation with Energy Function
AI总结：
本文提出基于能量函数的知识蒸馏方法（Energy KD），按能量分数对样本分类并自适应调整温度，以提升蒸馏判别能力，并引入高能量数据增强（HE-DA），在CIFAR-100-LT和ImageNet等数据集上显著优于现有方法。
摘要：
为了在真实工业应用中部署需要大量计算成本的最新计算机视觉技术，知识蒸馏方法（KDs）至关重要。现有的基于logit的知识蒸馏方法对数据集中的所有样本应用恒定的温度缩放，限制了对每个样本所蕴含知识的单独利用。在我们的方法中，我们根据能量分数将数据集分为两类（即低能量样本和高能量样本）。通过实验，我们确认低能量样本表现出高置信度分数，表示确定的预测，而高能量样本产生低置信度分数，表示不确定的预测。为了通过调整非目标类预测来蒸馏最优知识，我们对低能量样本应用更高的温度以生成更平滑的分布，对高能量样本应用更低的温度以实现更尖锐的分布。与先前的基于logit和基于特征的方法相比，我们基于能量的知识蒸馏（Energy KD）在各种数据集上取得了更好的性能。特别是，Energy KD在包含许多具有挑战性样本的CIFAR-100-LT和ImageNet数据集上显示出显著的改进。此外，我们提出了基于高能量的数据增强（HE-DA）以进一步提高性能。我们证明，仅增强数据集的一部分而非整个数据集可以实现更高的性能提升，这表明它可以用于资源受限的设备。据我们所知，本文是首次尝试在知识蒸馏和数据增强中利用能量函数，我们相信这将极大地促进未来的研究。
英文摘要：
To apply the latest computer vision techniques that require a large computational cost in real industrial applications, knowledge distillation methods (KDs) are essential. Existing logit-based KDs apply the constant temperature scaling to all samples in dataset, limiting the utilization of knowledge inherent in each sample individually. In our approach, we classify the dataset into two categories (i.e., low energy and high energy samples) based on their energy score. Through experiments, we have confirmed that low energy samples exhibit high confidence scores, indicating certain predictions, while high energy samples yield low confidence scores, meaning uncertain predictions. To distill optimal knowledge by adjusting non-target class predictions, we apply a higher temperature to low energy samples to create smoother distributions and a lower temperature to high energy samples to achieve sharper distributions. When compared to previous logit-based and feature-based methods, our energy-based KD (Energy KD) achieves better performance on various datasets. Especially, Energy KD shows significant improvements on CIFAR-100-LT and ImageNet datasets, which contain many challenging samples. Furthermore, we propose high energy-based data augmentation (HE-DA) for further improving the performance. We demonstrate that higher performance improvement could be achieved by augmenting only a portion of the dataset rather than the entire dataset, suggesting that it can be employed on resource-limited devices. To the best of our knowledge, this paper represents the first attempt to make use of energy function in knowledge distillation and data augmentation, and we believe it will greatly contribute to future research.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
