arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2110.12484cs.LGcs.DC
收藏

在保持性能的同时,使 DNN 模型能够突破内存限制进行大批量训练

Enabling Large Batch Size Training for DNN Models Beyond the Memory Limit While Maintaining Performance

Korea University(高丽大学);Georgia Institute of Technology(佐治亚理工学院);Nanyang Technological University(南洋理工大学)

XinYu Piao, DoangJoo Synn, JooYoung Park, Jong-Kook Kim

AI总结:

本文提出 Micro-Batch Processing(MBP),通过微批切分、顺序处理与基于梯度累积的损失归一化,使 DNN 在单设备内存受限情况下实现大批量训练并保持性能。

摘要:

近期的深度学习模型难以使用大批量规模进行训练,因为商用机器可能没有足够内存来同时容纳模型和大批量数据。批量大小是训练模型时使用的超参数之一,它依赖并受限于目标机器的内存容量,因为只有在模型加载后,批量才能放入剩余内存中。此外,数据项大小也是一个重要因素,因为如果每个数据项越大,能够放入剩余内存的批量大小就越小。本文提出一种名为 Micro-Batch Processing(MBP,微批处理)的方法来解决这一问题。该方法通过提供一种批处理方式来帮助深度学习模型训练,即将一个批量切分成能够放入剩余内存的大小,并按顺序处理它们。在分别处理这些小批量之后,采用一种基于梯度累积的损失归一化算法来保持性能。我们方法的目标是让深度学习模型能够使用超出系统内存容量的更大批量进行训练,而无需增加内存大小或使用多个设备(GPU)。

英文摘要:

Recent deep learning models are difficult to train using a large batch size, because commodity machines may not have enough memory to accommodate both the model and a large data batch size. The batch size is one of the hyper-parameters used in the training model, and it is dependent on and is limited by the target machine memory capacity because the batch size can only fit into the remaining memory after the model is uploaded. Moreover, the data item size is also an important factor because if each data item size is larger then the batch size that can fit into the remaining memory becomes smaller. This paper proposes a method called Micro-Batch Processing (MBP) to address this problem. This method helps deep learning models to train by providing a batch processing method that splits a batch into a size that can fit in the remaining memory and processes them sequentially. After processing the small batches individually, a loss normalization algorithm based on the gradient accumulation is used to maintain the performance. The purpose of our method is to allow deep learning models to train using larger batch sizes that exceed the memory capacity of a system without increasing the memory size or using multiple devices (GPUs).

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑