$100K 或 100 天:利用学术资源进行预训练时的权衡
$100K or 100 Days: Trade-offs when Pre-Training with Academic Resources
AI总结:
本文通过调查和基准测试,证明学术资源也能有效预训练模型,如用4个GPU在18天复制Pythia-1B,并提供成本效益分析以指导权衡。
摘要:
预训练以计算密集著称,而学术研究人员以资源匮乏著称。因此,人们普遍认为学者无法预训练模型。在本文中,我们试图澄清这一假设。我们首先调查学术研究人员以了解他们可用的计算资源,然后实证测量在此类资源上复制模型所需的时间。我们引入了一个基准来衡量在给定 GPU 上预训练模型的时间,并确定了最大化训练速度的理想设置。我们在多种模型和学术 GPU 上运行我们的基准,花费了 2,000 GPU 小时进行实验。我们的结果揭示了学术预训练的乐观前景:例如,尽管 Pythia-1B 最初是在 64 个 GPU 上训练 3 天,我们发现也可以使用相同的超参数在 GPU 天数减少 3 倍的情况下复制该模型:即在 4 个 GPU 上训练 18 天。最后,我们进行了成本效益分析,以帮助阐明价格与预训练时间之间的权衡。我们相信我们的基准将帮助学术研究人员进行需要在更多数据上训练更大模型的实验。我们在 https://github.com/apoorvkh/academic-pretraining 上完全发布了我们的代码库。
英文摘要:
Pre-training is notoriously compute-intensive and academic researchers are notoriously under-resourced. It is, therefore, commonly assumed that academics can't pre-train models. In this paper, we seek to clarify this assumption. We first survey academic researchers to learn about their available compute and then empirically measure the time to replicate models on such resources. We introduce a benchmark to measure the time to pre-train models on given GPUs and also identify ideal settings for maximizing training speed. We run our benchmark on a range of models and academic GPUs, spending 2,000 GPU-hours on our experiments. Our results reveal a brighter picture for academic pre-training: for example, although Pythia-1B was originally trained on 64 GPUs for 3 days, we find it is also possible to replicate this model (with the same hyper-parameters) in 3x fewer GPU-days: i.e. on 4 GPUs in 18 days. We conclude with a cost-benefit analysis to help clarify the trade-offs between price and pre-training time. We believe our benchmark will help academic researchers conduct experiments that require training larger models on more data. We fully release our codebase at: https://github.com/apoorvkh/academic-pretraining.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。