一个网络并不能统治一切:超越自监督学习中的手工架构
One Network Doesn't Rule Them All: Moving Beyond Handcrafted Architectures in Self-Supervised Learning
摘要:
当前关于自监督学习(SSL)的文献集中于开发学习目标,以在未标记数据上更有效地训练神经网络。典型的开发过程包括采用成熟的架构,例如,ResNet在ImageNet上演示,并使用它们来评估下游场景中新开发的目标。虽然方便,但这并没有考虑到架构的作用,这在监督学习文献中已被证明是至关重要的。在这项工作中,我们建立了广泛的经验证据表明,网络架构在SSL中起着重要的作用。我们对ResNet和MobileNet架构的100多种变体进行了大规模研究,并在SSL设置中的11个下游场景中对其进行了评估。我们表明,没有一个网络在所有场景中表现一致。在此基础上,我们建议不仅学习网络权重,而且在SSL制度的架构拓扑结构。我们表明,“自监督架构”优于流行的手工架构(ResNet 18和MobileNetV 2),同时在主要的图像分类基准测试(ImageNet-1 K,iNat 2021等)上与更大且计算量更大的ResNet 50竞争。我们的研究结果表明,现在是时候考虑超越SSL中的手工架构,并开始考虑将架构搜索纳入自监督学习目标。
英文摘要:
The current literature on self-supervised learning (SSL) focuses on developing learning objectives to train neural networks more effectively on unlabeled data. The typical development process involves taking well-established architectures, e.g., ResNet demonstrated on ImageNet, and using them to evaluate newly developed objectives on downstream scenarios. While convenient, this does not take into account the role of architectures which has been shown to be crucial in the supervised learning literature. In this work, we establish extensive empirical evidence showing that a network architecture plays a significant role in SSL. We conduct a large-scale study with over 100 variants of ResNet and MobileNet architectures and evaluate them across 11 downstream scenarios in the SSL setting. We show that there is no one network that performs consistently well across the scenarios. Based on this, we propose to learn not only network weights but also architecture topologies in the SSL regime. We show that "self-supervised architectures" outperform popular handcrafted architectures (ResNet18 and MobileNetV2) while performing competitively with the larger and computationally heavy ResNet50 on major image classification benchmarks (ImageNet-1K, iNat2021, and more). Our results suggest that it is time to consider moving beyond handcrafted architectures in SSL and start thinking about incorporating architecture search into self-supervised learning objectives.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。