arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2106.09785cs.CVcs.AIcs.LG
收藏

用于表示学习的高效自监督视觉Transformer

Efficient Self-supervised Vision Transformers for Representation Learning

Microsoft Research(微软研究院);Microsoft Cloud + AI(微软云计算与人工智能部门)

Chunyuan Li, Jianwei Yang, Pengchuan Zhang, Mei Gao, Bin Xiao, Xiyang Dai, Lu Yuan, Jianfeng Gao

摘要:

本文研究了两种技术,开发有效的自我监督视觉Transformer(EsViT)的视觉表示学习。首先,我们通过一个全面的实证研究表明,多阶段架构与稀疏的自我注意力可以显着降低建模的复杂性,但失去了捕捉图像区域之间的细粒度对应的能力的成本。其次,我们提出了一个新的区域匹配的预训练任务,它允许模型捕获细粒度的区域依赖关系,从而显着提高了学习的视觉表示的质量。我们的研究结果表明,结合这两种技术,EsViT在ImageNet线性探针评估中达到了81.3%的top-1,优于现有技术,吞吐量高出一个数量级。当转移到下游线性分类任务时,EsViT在18个数据集中的17个数据集上的表现优于其监督对手。代码和模型可在https://github.com/microsoft/esvit上公开获取

英文摘要:

This paper investigates two techniques for developing efficient self-supervised vision transformers (EsViT) for visual representation learning. First, we show through a comprehensive empirical study that multi-stage architectures with sparse self-attentions can significantly reduce modeling complexity but with a cost of losing the ability to capture fine-grained correspondences between image regions. Second, we propose a new pre-training task of region matching which allows the model to capture fine-grained region dependencies and as a result significantly improves the quality of the learned vision representations. Our results show that combining the two techniques, EsViT achieves 81.3% top-1 on the ImageNet linear probe evaluation, outperforming prior arts with around an order magnitude of higher throughput. When transferring to downstream linear classification tasks, EsViT outperforms its supervised counterpart on 17 out of 18 datasets. The code and models are publicly available: https://github.com/microsoft/esvit

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑