对比掩码自动编码器是更强的视觉学习器
Contrastive Masked Autoencoders are Stronger Vision Learners
AI总结：
针对掩码图像建模所学视觉表征判别力不足的问题，提出融合对比学习与掩码图像建模的自监督预训练方法CMAE，通过像素移位、特征解码器等设计提升表征质量，在多个视觉基准上取得SOTA结果。
摘要：
掩码图像建模（Masked image modeling, MIM）已在各类视觉任务上取得了可观的成果。然而，所学表征的判别能力有限，这表明要打造更强的视觉学习器仍有很长的路要走。为实现这一目标，我们提出了对比掩码自动编码器（Contrastive Masked Autoencoders, CMAE），这是一种全新的自监督预训练方法，用于学习更全面、性能更强的视觉表征。通过新颖的设计巧妙融合对比学习（contrastive learning, CL）与掩码图像模型（MIM），CMAE 发挥了二者各自的优势，所学习的表征同时具备强实例判别能力与局部感知能力。具体而言，CMAE 包含两个分支：在线分支是一个非对称编码器-解码器结构，动量分支是一个经动量更新的编码器。训练过程中，在线编码器从掩码图像的潜在表征中重建原始图像，以学习整体特征。输入完整图像的动量编码器通过与在线分支进行对比学习，增强特征的判别能力。为使 CL 与 MIM 相适配，CMAE 引入了两个全新组件，即用于生成合理正样本视图的像素移位模块，以及用于补充对比对特征的特征解码器。得益于这些新颖设计，CMAE 相较于对应的 MIM 方法，有效提升了表征质量与迁移性能。CMAE 在图像分类、语义分割和目标检测的高竞争力基准上取得了当前最优性能。值得注意的是，CMAE-Base 在 ImageNet 上达到 85.3% 的 top-1 准确率，在 ADE20k 上达到 52.5% 的 mIoU，分别超越此前最优结果 0.7% 和 1.8%。源代码已公开于 https://github.com/ZhichengHuang/CMAE。
英文摘要：
Masked image modeling (MIM) has achieved promising results on various vision tasks. However, the limited discriminability of learned representation manifests there is still plenty to go for making a stronger vision learner. Towards this goal, we propose Contrastive Masked Autoencoders (CMAE), a new self-supervised pre-training method for learning more comprehensive and capable vision representations. By elaboratively unifying contrastive learning (CL) and masked image model (MIM) through novel designs, CMAE leverages their respective advantages and learns representations with both strong instance discriminability and local perceptibility. Specifically, CMAE consists of two branches where the online branch is an asymmetric encoder-decoder and the momentum branch is a momentum updated encoder. During training, the online encoder reconstructs original images from latent representations of masked images to learn holistic features. The momentum encoder, fed with the full images, enhances the feature discriminability via contrastive learning with its online counterpart. To make CL compatible with MIM, CMAE introduces two new components, i.e. pixel shifting for generating plausible positive views and feature decoder for complementing features of contrastive pairs. Thanks to these novel designs, CMAE effectively improves the representation quality and transfer performance over its MIM counterpart. CMAE achieves the state-of-the-art performance on highly competitive benchmarks of image classification, semantic segmentation and object detection. Notably, CMAE-Base achieves $85.3\%$ top-1 accuracy on ImageNet and $52.5\%$ mIoU on ADE20k, surpassing previous best results by $0.7\%$ and $1.8\%$ respectively. The source code is publicly accessible at \url{https://github.com/ZhichengHuang/CMAE}.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
