arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2408.16947cs.LG
收藏

迁移学习缩放定律的实证研究

An Empirical Study of Scaling Laws for Transfer

Epoch AI

Matthew Barnett

AI总结:

本文通过实证研究提出含“迁移差距”项的缩放定律,揭示预训练与微调数据在不同分布间的成本效益权衡,为迁移学习效率测量和最优数据分配提供理论依据。

摘要:

我们对Transformer模型中的迁移学习缩放定律进行了一项有限的实证研究。具体而言,我们考察了一种包含“迁移差距”项的缩放定律,该迁移差距表示在一个分布上进行预训练对在另一个分布上优化下游性能的有效性。当迁移差距较小时,预训练是一种提升下游性能的经济高效策略。反之,当差距较大时,收集高质量的微调数据变得相对更具成本效益。将该缩放定律拟合到来自多样化数据集的实验中,揭示了不同分布之间迁移差距的显著差异。理论上,该缩放定律可以为最优数据分配策略提供依据,并突出下游数据稀缺如何成为性能瓶颈。我们的发现有助于以有原则的方式衡量迁移学习效率,并理解数据可用性如何影响模型能力。

英文摘要:

We present a limited empirical study of scaling laws for transfer learning in transformer models. More specifically, we examine a scaling law that incorporates a "transfer gap" term, indicating the effectiveness of pre-training on one distribution when optimizing for downstream performance on another distribution. When the transfer gap is low, pre-training is a cost-effective strategy for improving downstream performance. Conversely, when the gap is high, collecting high-quality fine-tuning data becomes relatively more cost effective. Fitting the scaling law to experiments from diverse datasets reveals significant variations in the transfer gap across distributions. In theory, the scaling law can inform optimal data allocation strategies and highlights how the scarcity of downstream data can bottleneck performance. Our findings contribute to a principled way to measure transfer learning efficiency and understand how data availability affects capabilities.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑