ReMem：面向有效知识蒸馏的预训练 Vision Transformers 互信息感知微调
ReMem: Mutual Information-Aware Fine-tuning of Pretrained Vision Transformers for Effective Knowledge Distillation
AI总结：
针对大规模预训练 ViT 强模型知识蒸馏效果下降的问题，提出互信息感知微调与重加权 MLP 块启发式方法，使小型学生模型能有效从最强预训练模型中受益。
摘要：
从预训练视觉表示模型进行知识蒸馏，是改进小型任务特定生产模型的有效方法。然而，当从大规模预训练的强模型中进行蒸馏时，这种知识转移的有效性会显著下降。在本文中，我们通过探索微调方法以实现更有效的知识转移，来解决预训练 Vision Transformers (ViTs) 的这一挑战。受互信息与蒸馏有效性之间联系的启发，我们提出在微调过程中采用互信息感知优化。对于此类优化效果减弱的小型或高度不平衡的下游数据集，我们引入了一种简单而有效的重加权 MLP 块的启发式方法。该方法受到我们观察的启发，即顶部 MLP 块是导致互信息损失的主要原因。我们的方法使小型学生模型能够从那些最强的预训练模型中受益。
英文摘要：
Knowledge distillation from pretrained visual representation models offers an effective approach to improve small, task-specific production models. However, the effectiveness of such knowledge transfer drops significantly when distilling from strong models that are pretrained in a large scale. In this paper, we address this challenge for pretrained Vision Transformers (ViTs) by exploring methods to fine-tune them for more effective knowledge transfer. Motivated by the connection between mutual information and distillation effectiveness, we propose to employ mutual information-aware optimization during finetuning. For small or highly-imbalanced downstream datasets where such optimization becomes less effective, we introduce a simple yet effective heuristic of reweighting MLP blocks. This approach is inspired by our observation that top MLP blocks are primarily responsible for mutual information loss. Our method enables small student models to benefit from those pretrained models among the strongest.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
