支持具有小数据集的深度神经网络的强缩放的论点
An argument in favor of strong scaling for deep neural networks with small datasets
摘要：
近年来，随着深度学习框架和大型数据集的普及，研究人员开始并行化他们的模型，以便更快地进行训练。这一点至关重要，因为他们通常会探索许多超参数，以便找到最适合其应用的超参数。这一过程很耗时，因此，加快培训可以提高生产率。许多研究人员采用的并行化深度学习模型的一种方法是基于弱缩放。随着新的GPU被添加到系统中，小批量的大小会增加。此外，已经提出了新的学习率时间表，以修复大的minibatch大小发生的优化问题。然而，在本文中，我们表明，最近的工作提供的建议不适用于缺乏大型数据集的模型。事实上，我们主张在这种情况下使用强大的扩展来实现可靠的性能。我们使用多达32个GPU对我们的方法进行了评估，结果表明，弱缩放不仅没有与顺序模型相同的准确性，而且在大多数情况下也无法收敛。同时，强缩放具有良好的可扩展性，同时具有与顺序实现完全相同的精度。
英文摘要：
In recent years, with the popularization of deep learning frameworks and large datasets, researchers have started parallelizing their models in order to train faster. This is crucially important, because they typically explore many hyperparameters in order to find the best ones for their applications. This process is time consuming and, consequently, speeding up training improves productivity. One approach to parallelize deep learning models followed by many researchers is based on weak scaling. The minibatches increase in size as new GPUs are added to the system. In addition, new learning rates schedules have been proposed to fix optimization issues that occur with large minibatch sizes. In this paper, however, we show that the recommendations provided by recent work do not apply to models that lack large datasets. In fact, we argument in favor of using strong scaling for achieving reliable performance in such cases. We evaluated our approach with up to 32 GPUs and show that weak scaling not only does not have the same accuracy as the sequential model, it also fails to converge most of time. Meanwhile, strong scaling has good scalability while having exactly the same accuracy of a sequential implementation.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
