arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2310.16764cs.CVcs.LGcs.NE
收藏

ConvNets在大规模下匹配Vision Transformers

ConvNets Match Vision Transformers at Scale

Google DeepMind(谷歌DeepMind)

Samuel L. Smith, Andrew Brock, Leonard Berrada, Soham De

AI总结:

针对ConvNets在大规模数据上无法与Vision Transformers竞争的观点,本文通过在JFT-4B数据集上预训练NFNet模型家族发现其遵循对数-对数缩放定律,在同等计算预算下匹配了ViT性能,并在ImageNet微调后达到90.4%的Top-1准确率。

摘要:

许多研究人员认为,ConvNets在小型或中等规模数据集上表现良好,但在访问网络规模数据集时无法与Vision Transformers竞争。我们通过评估在JFT-4B(一个常用于训练基础模型的大型图像标注数据集)上预训练的高性能ConvNet架构来挑战这一观点。我们考虑了0.4k至110k TPU-v4核心计算小时的预训练计算预算,并训练了一系列来自NFNet模型家族、深度和宽度递增的网络。我们观察到保留损失与计算预算之间存在对数-对数缩放定律。在ImageNet上微调后,NFNets在可比计算预算下匹配了Vision Transformers的报告性能。我们最强的微调模型实现了90.4%的Top-1准确率。

英文摘要:

Many researchers believe that ConvNets perform well on small or moderately sized datasets, but are not competitive with Vision Transformers when given access to datasets on the web-scale. We challenge this belief by evaluating a performant ConvNet architecture pre-trained on JFT-4B, a large labelled dataset of images often used for training foundation models. We consider pre-training compute budgets between 0.4k and 110k TPU-v4 core compute hours, and train a series of networks of increasing depth and width from the NFNet model family. We observe a log-log scaling law between held out loss and compute budget. After fine-tuning on ImageNet, NFNets match the reported performance of Vision Transformers with comparable compute budgets. Our strongest fine-tuned model achieves a Top-1 accuracy of 90.4%.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑