arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2212.14034cs.CLcs.LG
收藏

填鸭式:一天内在单个GPU上训练语言模型

Cramming: Training a Language Model on a Single GPU in One Day

University of Maryland, College Park(马里兰大学帕克分校)

Jonas Geiping, Tom Goldstein

摘要:

语言建模的最新趋势集中在通过扩展来提高性能,并导致大多数研究人员和实践者无法获得训练语言模型的环境。虽然社区中的大多数人都在问如何推动极限计算,但我们提出了相反的问题:我们在一天内可以用一个GPU走多远?我们研究了基于transformer的语言模型的下游性能,该模型在单个消费者GPU上使用掩蔽语言建模进行了一天的完全从头开始训练。除了重新分析这种情况下预训练管道的几乎所有组件,并提供一个性能接近BERT的修改后的管道外,我们还研究了为什么缩小是困难的,以及哪些修改实际上提高了这种情况下的性能。我们提供的证据表明,即使在这种约束的设置,性能密切遵循缩放定律在大型计算设置中观察到的。通过缩放定律的镜头,我们对最近的一系列训练和架构改进进行了分类,并讨论了它们在有限计算环境中的优点和实用性(或缺乏实用性)。

英文摘要:

Recent trends in language modeling have focused on increasing performance through scaling, and have resulted in an environment where training language models is out of reach for most researchers and practitioners. While most in the community are asking how to push the limits of extreme computation, we ask the opposite question: How far can we get with a single GPU in just one day? We investigate the downstream performance achievable with a transformer-based language model trained completely from scratch with masked language modeling for a single day on a single consumer GPU. Aside from re-analyzing nearly all components of the pretraining pipeline for this scenario and providing a modified pipeline with performance close to BERT, we investigate why scaling down is hard, and which modifications actually improve performance in this scenario. We provide evidence that even in this constrained setting, performance closely follows scaling laws observed in large-compute settings. Through the lens of scaling laws, we categorize a range of recent improvements to training and architecture and discuss their merit and practical applicability (or lack thereof) for the limited compute setting.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑