ViC-MAE：基于对比掩码自编码器的图像与视频自监督表示学习
ViC-MAE: Self-Supervised Representation Learning from Images and Video with Contrastive Masked Autoencoders
AI总结：
本文提出 ViC-MAE，将 MAE 重建与跨图像和视频帧的对比学习相结合，在 ImageNet-1k 和 Something-something-v2 等基准上取得领先的自监督迁移学习性能。
摘要：
我们提出 ViC-MAE，这是一种结合了掩码自编码器（MAE）与对比学习的模型。ViC-MAE 的训练方式是：首先在 MAE 重建损失下学习局部表示，再通过对这些局部表示进行池化获得全局特征，并在图像和视频帧之间利用该全局特征进行对比学习。我们表明，在 ViC-MAE 下学习到的视觉表示能够很好地泛化到视频和图像分类任务。特别是，与最近提出的 OmniMAE 相比，ViC-MAE 在 ImageNet-1k 上从视频到图像的迁移学习性能达到了当前最优水平：在相同数据上训练时，top-1 准确率达到 86%（绝对提升 +1.3%）；在使用额外数据训练时，top-1 准确率达到 87.1%（绝对提升 +2.4%）。同时，ViC-MAE 在视频基准上也优于大多数其他方法，在具有挑战性的 Something-something-v2 视频基准上取得了 75.9% 的 top-1 准确率。当在来自多种数据集组合的视频和图像上进行训练时，我们的方法在视频和图像分类基准之间保持了均衡的迁移学习性能，仅以微弱差距落后于最佳监督方法。
英文摘要：
We propose ViC-MAE, a model that combines both Masked AutoEncoders (MAE) and contrastive learning. ViC-MAE is trained using a global featured obtained by pooling the local representations learned under an MAE reconstruction loss and leveraging this representation under a contrastive objective across images and video frames. We show that visual representations learned under ViC-MAE generalize well to both video and image classification tasks. Particularly, ViC-MAE obtains state-of-the-art transfer learning performance from video to images on Imagenet-1k compared to the recently proposed OmniMAE by achieving a top-1 accuracy of 86% (+1.3% absolute improvement) when trained on the same data and 87.1% (+2.4% absolute improvement) when training on extra data. At the same time ViC-MAE outperforms most other methods on video benchmarks by obtaining 75.9% top-1 accuracy on the challenging Something something-v2 video benchmark . When training on videos and images from a diverse combination of datasets, our method maintains a balanced transfer-learning performance between video and image classification benchmarks, coming only as a close second to the best supervised method.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
