混合蒸馏：连接掩码自编码器与对比学习器
Hybrid Distillation: Connecting Masked Autoencoders with Contrastive Learners
AI总结：
本文提出混合蒸馏策略，结合监督/CL教师和MIM教师的优势，通过模仿token关系和蒸馏特征图，提升模型的判别性和多样性，并在多个基准上取得优越性能。
摘要：
表征学习一直在从传统的监督训练向对比学习（CL）和掩码图像建模（MIM）演进。先前的研究已经证明了它们在特定场景下的优缺点，即CL和监督预训练擅长捕捉更长距离的全局模式并实现更好的特征判别，而MIM能够在所有transformer层中引入更多局部和多样化的注意力。在本文中，我们探索如何获得一个结合它们优势的模型。我们首先检查了先前的特征蒸馏和掩码特征重建方法，并识别了它们的局限性。我们发现，它们日益增加的多样性主要源于非对称设计，但这些设计反过来可能损害判别能力。为了更好地同时获得判别性和多样性，我们提出了一种简单但有效的混合蒸馏策略，该策略同时利用监督/CL教师和MIM教师来联合指导学生模型。Hybrid Distill模仿MIM教师的token关系以缓解注意力崩溃，并蒸馏监督/CL教师的特征图以实现判别。此外，还利用了渐进式冗余token掩码策略来降低蒸馏成本并避免陷入局部最优。实验结果证明，Hybrid Distill能够在不同基准上实现优越的性能。
英文摘要：
Representation learning has been evolving from traditional supervised training to Contrastive Learning (CL) and Masked Image Modeling (MIM). Previous works have demonstrated their pros and cons in specific scenarios, i.e., CL and supervised pre-training excel at capturing longer-range global patterns and enabling better feature discrimination, while MIM can introduce more local and diverse attention across all transformer layers. In this paper, we explore how to obtain a model that combines their strengths. We start by examining previous feature distillation and mask feature reconstruction methods and identify their limitations. We find that their increasing diversity mainly derives from the asymmetric designs, but these designs may in turn compromise the discrimination ability. In order to better obtain both discrimination and diversity, we propose a simple but effective Hybrid Distillation strategy, which utilizes both the supervised/CL teacher and the MIM teacher to jointly guide the student model. Hybrid Distill imitates the token relations of the MIM teacher to alleviate attention collapse, as well as distills the feature maps of the supervised/CL teacher to enable discrimination. Furthermore, a progressive redundant token masking strategy is also utilized to reduce the distilling costs and avoid falling into local optima. Experiment results prove that Hybrid Distill can achieve superior performance on different benchmarks.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
