MST: Masked Self-Supervised Transformer for Visual Representation
Institute of Automation, CAS(中国科学院自动化研究所);University of Chinese Academy of Sciences(中国科学院大学);SenseTime Research(商汤科技研究院);University of California, Los Angeles(加利福尼亚大学洛杉矶分校)
Zhaowen Li, Zhiyang Chen, Fan Yang, Wei Li, Yousong Zhu, Chaoyang Zhao, Rui Deng, Liwei Wu, Rui Zhao, Ming Tang, Jinqiao Wang
Transformer已被广泛应用于自然语言处理(NLP)中的自监督预训练,并取得了巨大的成功。然而,在视觉自监督学习中,它还没有得到充分的探索。同时,以往的方法只从全局的角度考虑高层特征和学习表示,可能无法将其转移到关注局部特征的下游密集预测任务中。在本文中,我们提出了一种新的Masked自监督Transformer方法命名为MST,它可以显式地捕获图像的局部上下文,同时保留全局语义信息。具体而言,受自然语言处理中的MLM(Masked Language Modeling)思想的启发,提出了一种基于多头自注意映射的Masked token策略,该策略在不破坏自监督学习关键结构的前提下,动态地屏蔽局部块的部分token。更重要的是,屏蔽令牌连同剩余令牌一起由全局图像解码器进一步恢复,该全局图像解码器保留了图像的空间信息,并且对下游密集预测任务更友好。在多个数据集上的实验证明了该方法的有效性和通用性。例如,MST使用DeiT-S仅使用300个epoch预训练通过线性评估实现了76.9%的Top-1准确率,这比具有相同epoch的监督方法高出0.4%,比其可比变体DINO高出1.0%。对于密集预测任务,MST还在MS COCO目标检测上实现了42.7%的mAP,在Cityscapes分割上实现了74.04%的mIoU,仅使用100个epoch的预训练。
英文摘要:
Transformer has been widely used for self-supervised pre-training in Natural Language Processing (NLP) and achieved great success. However, it has not been fully explored in visual self-supervised learning. Meanwhile, previous methods only consider the high-level feature and learning representation from a global perspective, which may fail to transfer to the downstream dense prediction tasks focusing on local features. In this paper, we present a novel Masked Self-supervised Transformer approach named MST, which can explicitly capture the local context of an image while preserving the global semantic information. Specifically, inspired by the Masked Language Modeling (MLM) in NLP, we propose a masked token strategy based on the multi-head self-attention map, which dynamically masks some tokens of local patches without damaging the crucial structure for self-supervised learning. More importantly, the masked tokens together with the remaining tokens are further recovered by a global image decoder, which preserves the spatial information of the image and is more friendly to the downstream dense prediction tasks. The experiments on multiple datasets demonstrate the effectiveness and generality of the proposed method. For instance, MST achieves Top-1 accuracy of 76.9% with DeiT-S only using 300-epoch pre-training by linear evaluation, which outperforms supervised methods with the same epoch by 0.4% and its comparable variant DINO by 1.0\%. For dense prediction tasks, MST also achieves 42.7% mAP on MS COCO object detection and 74.04% mIoU on Cityscapes segmentation only with 100-epoch pre-training.
Efficient Self-supervised Vision Pretraining with Local Masked Reconstruction
King Abdullah University of Science and Technology(阿卜杜拉国王科技大学);Monash University(莫纳什大学);Nile University(尼罗河大学);Nanyang Technological University(南洋理工大学)
Self-Supervised Learning with Masked Image Modeling for Teeth Numbering, Detection of Dental Restorations, and Instance Segmentation in Dental Panoramic Radiographs
MV-JAR: Masked Voxel Jigsaw and Reconstruction for LiDAR-Based Self-Supervised Pre-Training
The Chinese University of Hong Kong(香港中文大学);Shanghai AI Laboratory(上海人工智能实验室);The University of Hong Kong(香港大学);Carnegie Mellon University(卡内基梅隆大学)
本文介绍了基于LiDAR的自监督预训练的掩蔽体素拼图和重建(MV-Jigsaw and Reconstruction)方法,以及在Waymo数据集上精心设计的数据高效的3D目标检测基准。受下游3D对象检测器中的场景-体素-点层次结构的启发,我们设计了考虑场景中的体素分布和体素内的局部点分布的掩蔽和重建策略。我们采用了一种反向Fursed-Voxel-Sampling策略来解决LiDAR点的不均匀分布,并提出了MV-Voxel,它结合了两种技术来建模上述分布,从而获得了优异的性能。我们的实验揭示了以前数据高效实验的局限性,这些实验均匀地对每个LiDAR序列中不同数据比例的微调分裂进行采样,导致分裂之间的数据多样性相似。为了解决这个问题,我们提出了一个新的基准,该基准对场景序列进行采样以进行各种微调分割,确保足够的模型收敛并提供对预训练方法的更准确评估。在Waymo基准测试和KITTI数据集上的实验表明,MV-SVM在各种数据尺度上都能持续显著地提高3D检测性能,与从头开始训练相比,mAPH提高了6.3%。代码和基准将在https://github.com/SmartBot-PJLab/MV-JAR上提供。
Boosting Semi-Supervised Medical Image Segmentation via Masked Image Consistency and Discrepancy Learning
School of Information and Communication Engineering, Beijing University of Posts and Telecommunications(北京邮电大学信息与通信工程学院);School of Artificial Intelligence, Beijing University of Posts and Telecommunications(北京邮电大学人工智能学院);School of Computer Science and Engineering, Southeast University(东南大学计算机科学与工程学院)
RI-MAE: Rotation-Invariant Masked AutoEncoders for Self-Supervised Point Cloud Representation Learning
South China University of Technology(华南理工大学);The University of Adelaide(阿德莱德大学);The University of Western Australia(西澳大学);The University of Sydney(悉尼大学)
Understanding Self-Supervised Pretraining with Part-Aware Representation Learning
Key Lab of High Confidence Software Technologies (Peking University), Ministry of Education, China(高可信软件技术教育部重点实验室(北京大学));School of Computer Science, Peking University(北京大学计算机学院);School of AI, Peking University(北京大学人工智能学院);University of Hong Kong(香港大学);School of EIC, Huazhong University of Science & Technology(华中科技大学电子信息与通信学院);Baidu(百度);UC Berkeley(加州大学伯克利分校)
Exploring Self-Supervised Learning with U-Net Masked Autoencoders and EfficientNet-B7 for Improved Gastrointestinal Abnormality Classification in Video Capsule Endoscopy
International Institute of Information Technology, Bangalore(国际信息技术学院,班加罗尔);Indian Institute of Technology, Hyderabad(印度理工学院,海得拉巴)
Masked Next-Scale Prediction for Self-supervised Scene Text Recognition
Nankai University(南开大学);Institute of Information Engineering, Chinese Academy of Sciences(中国科学院信息工程研究所);Department of Automation and BNRist, Tsinghua University(清华大学自动化系和清华大学脑科学与智能技术研究院)
Sparse Anatomical Prompt Semi-Supervised Learning with Masked Image Modeling for CBCT Tooth Segmentation
Institute of Innovative Research, Tokyo Institute of Technology(东京工业大学创新研究院);Stomatological Hospital of Chongqing Medical University(重庆医科大学附属口腔医院);School of Mechanical Engineering, Zhejiang University(浙江大学机械工程学院)
HMSViT: A Hierarchical Masked Self-Supervised Vision Transformer for Corneal Nerve Segmentation and Diabetic Neuropathy Diagnosis
Department of Computing and Mathematics, Manchester Metropolitan University(计算与数学系,曼彻斯特 Metropolitan 大学);Department of Eye and Vision Sciences, University of Liverpool(眼科学与视觉科学系,利物浦大学);Faculty of Biology, Medicine and Health, University of Manchester(生物、医学与健康学院,曼彻斯特大学);Department of Medicine, Weill Cornell Medicine-Qatar(医学系,Weill Cornell Medicine-Qatar)
Domain Invariant Masked Autoencoders for Self-supervised Learning from Multi-domains
WuHan University(武汉大学);Zhejiang University(浙江大学);The University of Sydney(悉尼大学);SenseTime Research(商汤科技研究院);Qing Yuan Research Institute(清源研究院);Shanghai Jiao Tong University(上海交通大学)