arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2106.05656cs.CV
收藏

MST:用于视觉表示的掩蔽自监督Transformer

MST: Masked Self-Supervised Transformer for Visual Representation

Institute of Automation, CAS(中国科学院自动化研究所);University of Chinese Academy of Sciences(中国科学院大学);SenseTime Research(商汤科技研究院);University of California, Los Angeles(加利福尼亚大学洛杉矶分校)

Zhaowen Li, Zhiyang Chen, Fan Yang, Wei Li, Yousong Zhu, Chaoyang Zhao, Rui Deng, Liwei Wu, Rui Zhao, Ming Tang, Jinqiao Wang

摘要:

Transformer已被广泛应用于自然语言处理(NLP)中的自监督预训练,并取得了巨大的成功。然而,在视觉自监督学习中,它还没有得到充分的探索。同时,以往的方法只从全局的角度考虑高层特征和学习表示,可能无法将其转移到关注局部特征的下游密集预测任务中。在本文中,我们提出了一种新的Masked自监督Transformer方法命名为MST,它可以显式地捕获图像的局部上下文,同时保留全局语义信息。具体而言,受自然语言处理中的MLM(Masked Language Modeling)思想的启发,提出了一种基于多头自注意映射的Masked token策略,该策略在不破坏自监督学习关键结构的前提下,动态地屏蔽局部块的部分token。更重要的是,屏蔽令牌连同剩余令牌一起由全局图像解码器进一步恢复,该全局图像解码器保留了图像的空间信息,并且对下游密集预测任务更友好。在多个数据集上的实验证明了该方法的有效性和通用性。例如,MST使用DeiT-S仅使用300个epoch预训练通过线性评估实现了76.9%的Top-1准确率,这比具有相同epoch的监督方法高出0.4%,比其可比变体DINO高出1.0%。对于密集预测任务,MST还在MS COCO目标检测上实现了42.7%的mAP,在Cityscapes分割上实现了74.04%的mIoU,仅使用100个epoch的预训练。

英文摘要:

Transformer has been widely used for self-supervised pre-training in Natural Language Processing (NLP) and achieved great success. However, it has not been fully explored in visual self-supervised learning. Meanwhile, previous methods only consider the high-level feature and learning representation from a global perspective, which may fail to transfer to the downstream dense prediction tasks focusing on local features. In this paper, we present a novel Masked Self-supervised Transformer approach named MST, which can explicitly capture the local context of an image while preserving the global semantic information. Specifically, inspired by the Masked Language Modeling (MLM) in NLP, we propose a masked token strategy based on the multi-head self-attention map, which dynamically masks some tokens of local patches without damaging the crucial structure for self-supervised learning. More importantly, the masked tokens together with the remaining tokens are further recovered by a global image decoder, which preserves the spatial information of the image and is more friendly to the downstream dense prediction tasks. The experiments on multiple datasets demonstrate the effectiveness and generality of the proposed method. For instance, MST achieves Top-1 accuracy of 76.9% with DeiT-S only using 300-epoch pre-training by linear evaluation, which outperforms supervised methods with the same epoch by 0.4% and its comparable variant DINO by 1.0\%. For dense prediction tasks, MST also achieves 42.7% mAP on MS COCO object detection and 74.04% mIoU on Cityscapes segmentation only with 100-epoch pre-training.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑