MimCo：使用对比教师的掩码图像建模预训练
MimCo: Masked Image Modeling Pre-training with Contrastive Teacher
AI总结：
本文提出 MimCo 两阶段预训练框架，以对比学习模型为教师并结合 patch 级与图像级重建目标，显著增强 MIM 表征可分性及下游迁移性能。
摘要：
近期，掩码图像建模（MIM）在自监督学习（SSL）中受到广泛关注，它要求目标模型恢复输入图像中被掩码的部分。尽管基于 MIM 的预训练方法在迁移到许多下游任务时取得了新的最先进性能，但可视化结果表明，其学习到的表征可分性较差，尤其是与基于对比学习预训练所得的表征相比。这促使我们思考，能否进一步提升 MIM 预训练表征的线性可分性，从而改善预训练性能。由于 MIM 和对比学习往往采用不同的数据增强和训练策略，将这两个代理任务结合起来并非易事。在本工作中，我们提出了一个新颖且灵活的预训练框架，名为 MimCo，它通过两阶段预训练将 MIM 与对比学习结合起来。具体而言，MimCo 将一个预训练的对比学习模型作为教师模型，并使用两类学习目标进行预训练：patch 级和图像级重建损失。在下游任务上进行的大量迁移实验证明了我们 MimCo 预训练框架的优越性能。以 ViT-S 为例，当使用预训练的 MoCov3-ViT-S 作为教师模型时，MimCo 仅需 100 个 epoch 的预训练，即可在 Imagenet-1K 上达到 82.53% 的 top-1 微调准确率，优于最先进的自监督学习同类方法。
英文摘要：
Recent masked image modeling (MIM) has received much attention in self-supervised learning (SSL), which requires the target model to recover the masked part of the input image. Although MIM-based pre-training methods achieve new state-of-the-art performance when transferred to many downstream tasks, the visualizations show that the learned representations are less separable, especially compared to those based on contrastive learning pre-training. This inspires us to think whether the linear separability of MIM pre-trained representation can be further improved, thereby improving the pre-training performance. Since MIM and contrastive learning tend to utilize different data augmentations and training strategies, combining these two pretext tasks is not trivial. In this work, we propose a novel and flexible pre-training framework, named MimCo, which combines MIM and contrastive learning through two-stage pre-training. Specifically, MimCo takes a pre-trained contrastive learning model as the teacher model and is pre-trained with two types of learning targets: patch-level and image-level reconstruction losses. Extensive transfer experiments on downstream tasks demonstrate the superior performance of our MimCo pre-training framework. Taking ViT-S as an example, when using the pre-trained MoCov3-ViT-S as the teacher model, MimCo only needs 100 epochs of pre-training to achieve 82.53% top-1 finetuning accuracy on Imagenet-1K, which outperforms the state-of-the-art self-supervised learning counterparts.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
