arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2110.02095cs.LGcs.AIcs.CVstat.ML
收藏

探索大规模预训练的极限

Exploring the Limits of Large Scale Pre-training

Google Research(谷歌研究院)

Samira Abnar, Mostafa Dehghani, Behnam Neyshabur, Hanie Sedghi

摘要:

大规模机器学习的最新发展表明,通过适当地扩大数据、模型大小和训练时间,人们可能会观察到预训练的改进将有利地转移到大多数下游任务。在这项工作中,我们系统地研究这种现象,并建立,因为我们增加上游的准确性,下游任务的性能饱和。特别是,我们研究了视觉Transformer,MLP混合器和ResNets的4800多个实验,参数数量从1000万到100亿不等,在最大规模的可用图像数据(JFT,ImageNet21 K)上进行训练,并在20多个下游图像识别任务上进行评估。我们提出了一个下游性能模型,反映了饱和现象,并捕捉上游和下游任务的性能的非线性关系。深入研究,以了解引起这些现象的原因,我们表明,我们观察到的饱和行为是密切相关的方式表示通过模型的层演变。我们展示了一个更极端的场景,其中上游和下游的性能相互矛盾。也就是说,为了获得更好的下游性能,我们需要损害上游精度。

英文摘要:

Recent developments in large-scale machine learning suggest that by scaling up data, model size and training time properly, one might observe that improvements in pre-training would transfer favorably to most downstream tasks. In this work, we systematically study this phenomena and establish that, as we increase the upstream accuracy, the performance of downstream tasks saturates. In particular, we investigate more than 4800 experiments on Vision Transformers, MLP-Mixers and ResNets with number of parameters ranging from ten million to ten billion, trained on the largest scale of available image data (JFT, ImageNet21K) and evaluated on more than 20 downstream image recognition tasks. We propose a model for downstream performance that reflects the saturation phenomena and captures the nonlinear relationship in performance of upstream and downstream tasks. Delving deeper to understand the reasons that give rise to these phenomena, we show that the saturation behavior we observe is closely related to the way that representations evolve through the layers of the models. We showcase an even more extreme scenario where performance on upstream and downstream are at odds with each other. That is, to have a better downstream performance, we need to hurt upstream accuracy.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑