arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2104.02057cs.CVcs.LG
收藏

自我监督型视觉Transformer训练的实证研究

An Empirical Study of Training Self-Supervised Vision Transformers

Facebook AI Research (FAIR)(脸书人工智能研究院)

Xinlei Chen, Saining Xie, Kaiming He

摘要:

本文没有描述一种新方法。相反,鉴于计算机视觉的最新进展,它研究了一个简单的,增量的,但必须知道的基线:视觉Transformer(ViT)的自监督学习。虽然标准卷积网络的训练方法已经非常成熟和强大,但ViT的方法还有待建立,特别是在训练变得更具挑战性的自监督场景中。在这项工作中,我们回到基础知识,并研究了几个基本组成部分对训练自我监督ViT的影响。我们观察到,不稳定性是降低准确性的主要问题,并且它可以被表面上良好的结果所隐藏。我们发现,这些结果确实是部分失败的,他们可以得到改善时,培训更加稳定。我们在MoCo v3和其他几个自监督框架中对ViT结果进行了基准测试,并在各个方面进行了消融。我们讨论了目前的积极证据以及挑战和开放的问题。我们希望这项工作将为未来的研究提供有用的数据点和经验。

英文摘要:

This paper does not describe a novel method. Instead, it studies a straightforward, incremental, yet must-know baseline given the recent progress in computer vision: self-supervised learning for Vision Transformers (ViT). While the training recipes for standard convolutional networks have been highly mature and robust, the recipes for ViT are yet to be built, especially in the self-supervised scenarios where training becomes more challenging. In this work, we go back to basics and investigate the effects of several fundamental components for training self-supervised ViT. We observe that instability is a major issue that degrades accuracy, and it can be hidden by apparently good results. We reveal that these results are indeed partial failure, and they can be improved when training is made more stable. We benchmark ViT results in MoCo v3 and several other self-supervised frameworks, with ablations in various aspects. We discuss the currently positive evidence as well as challenges and open questions. We hope that this work will provide useful data points and experience for future research.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑