大规模语言建模:4小时内汇聚40GB文本
Large Scale Language Modeling: Converging on 40GB of Text in Four Hours
摘要:
最近的工作已经展示了如何在大型图像数据集上快速训练卷积神经网络(CNN),然后将从这些模型中获得的知识转移到各种任务中。继[Radford 2017]之后,在这项工作中,我们展示了用于自然语言任务的递归神经网络(RNN)的类似可扩展性和传输。通过利用混合精度算法和分布在128个NVIDIA Tesla V100 GPU上的32k批量大小,我们能够在4小时内训练字符级4096维乘法LSTM(mLSTM),用于40 GB Amazon Reviews数据集的3个epoch的无监督文本重建。这个运行时间与以前的工作相比毫不逊色,以前的工作需要一个月的时间来训练相同的大小和配置,一个时期在相同的数据集。收敛大批量RNN模型可能具有挑战性。最近的工作建议缩放的学习率作为批量大小的函数,但我们发现,简单地缩放的学习率作为批量大小的函数导致要么显着恶化收敛或立即发散这个问题。我们提供了一个学习率调度,使我们的模型能够收敛到32k的批量大小。由于我们的模型在几个小时内就收敛到Amazon Reviews数据集上,并且我们的计算需求是128个Tesla V100 GPU,虽然数量很大,但已经商用,因此这项工作为大多数商业应用和深度学习研究人员提供了大规模无监督NLP训练。一个模型可以在一夜之间在大多数公共或私有文本数据集上进行训练。
英文摘要:
Recent work has shown how to train Convolutional Neural Networks (CNNs) rapidly on large image datasets, then transfer the knowledge gained from these models to a variety of tasks. Following [Radford 2017], in this work, we demonstrate similar scalability and transfer for Recurrent Neural Networks (RNNs) for Natural Language tasks. By utilizing mixed precision arithmetic and a 32k batch size distributed across 128 NVIDIA Tesla V100 GPUs, we are able to train a character-level 4096-dimension multiplicative LSTM (mLSTM) for unsupervised text reconstruction over 3 epochs of the 40 GB Amazon Reviews dataset in four hours. This runtime compares favorably with previous work taking one month to train the same size and configuration for one epoch over the same dataset. Converging large batch RNN models can be challenging. Recent work has suggested scaling the learning rate as a function of batch size, but we find that simply scaling the learning rate as a function of batch size leads either to significantly worse convergence or immediate divergence for this problem. We provide a learning rate schedule that allows our model to converge with a 32k batch size. Since our model converges over the Amazon Reviews dataset in hours, and our compute requirement of 128 Tesla V100 GPUs, while substantial, is commercially available, this work opens up large scale unsupervised NLP training to most commercial applications and deep learning researchers. A model can be trained over most public or private text datasets overnight.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。