arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2102.01293cs.LG
收藏

传递的缩放定律

Scaling Laws for Transfer

OpenAI;Johns Hopkins University(约翰斯·霍普金斯大学)

Danny Hernandez, Jared Kaplan, Tom Henighan, Sam McCandlish

摘要:

我们研究经验缩放定律的迁移学习分布之间的无监督,微调设置。当我们在一个固定大小的数据集上从头开始训练越来越大的神经网络时,它们最终会变得数据有限,并停止提高性能(交叉熵损失)。当我们对在大型语言数据集上预训练的模型做同样的事情时,性能增益的斜率只是降低而不是变为零。我们通过确定相同大小的Transformer在从头开始训练时需要多少数据才能实现相同的损失来计算从预训练“传输”的有效数据。换句话说,我们专注于数据单元,而其他一切都是固定的。我们发现,有效的数据传输描述以及在低数据制度的幂律参数计数和微调数据集的大小。我们相信这些幂律中的指数对应于模型的一般性和分布的接近性(在定向而非对称意义上)的度量。我们发现预训练有效地增加了微调数据集的大小。传输与整体性能一样,在参数、数据和计算方面可预测地扩展。

英文摘要:

We study empirical scaling laws for transfer learning between distributions in an unsupervised, fine-tuning setting. When we train increasingly large neural networks from-scratch on a fixed-size dataset, they eventually become data-limited and stop improving in performance (cross-entropy loss). When we do the same for models pre-trained on a large language dataset, the slope in performance gains is merely reduced rather than going to zero. We calculate the effective data "transferred" from pre-training by determining how much data a transformer of the same size would have required to achieve the same loss when training from scratch. In other words, we focus on units of data while holding everything else fixed. We find that the effective data transferred is described well in the low data regime by a power-law of parameter count and fine-tuning dataset size. We believe the exponents in these power-laws correspond to measures of the generality of a model and proximity of distributions (in a directed rather than symmetric sense). We find that pre-training effectively multiplies the fine-tuning dataset size. Transfer, like overall performance, scales predictably in terms of parameters, data, and compute.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑