通过置信度引导的数据增强改进未知协变量偏移下的知识蒸馏
Improving Knowledge Distillation Under Unknown Covariate Shift Through Confidence-Guided Data Augmentation
AI总结：
针对未知协变量偏移下知识蒸馏效果受限的问题，提出置信度引导的扩散数据增强策略，经实验验证可提升多数据集上的相关准确率与虚假特征得分。
摘要：
在各类领域中，基于大规模数据集训练的大型基础模型展现出强大的零样本能力。当数据和模型规模受限时，知识蒸馏已成为将基础模型的知识迁移至小型学生网络的成熟工具。然而，蒸馏的效果常受限于训练数据覆盖范围有限的问题，这会导致训练数据与测试数据之间出现协变量偏移，进而使学生网络利用虚假特征甚至产生捷径学习。为解决该问题，本文提出一种新颖的基于扩散的数据增强策略，通过最大化教师模型与学生模型之间的分歧生成图像，有效构建学生网络难以应对的挑战性样本，从而缓解协变量偏移问题。实验表明，与最先进的基于扩散的数据增强基线方法相比，本文方法在样本平均准确率上排名第一或第二，且在协变量偏移下，可提升CelebA-HQ、SpuCo Birds和BAR上的最差组准确率与平均组准确率，以及Spurious ImageNet上的虚假特征得分。
英文摘要：
Large foundation models trained on extensive datasets demonstrate strong zero-shot capabilities in various domains. Knowledge distillation has become an established tool for transferring knowledge from foundation models to small student networks when data and model size are constrained. However, the efficacy of distillation is often hampered by limited training data coverage. This can result in a covariate shift between training and test data which in turn can lead the student to exploit spurious features or even shortcut learning. We address this problem by introducing a novel diffusion-based data augmentation strategy that generates images by maximizing the disagreement between the teacher and the student, effectively creating challenging samples that the student struggles with, thus mitigating the problem of covariate shift. Experiments demonstrate that, compared to state-of-the-art diffusion-based data augmentation baselines, our approach is best or second-best in sample mean accuracy and improves the worst group and mean group accuracy on CelebA-HQ, SpuCo Birds and BAR as well as the spurious score on Spurious ImageNet under covariate shift.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
