arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2106.00221cs.LGcs.AI
收藏

面向大批量训练的并发对抗学习

Concurrent Adversarial Learning for Large-Batch Training

National University of Singapore(新加坡国立大学);University of California, Los Angeles(加利福尼亚大学洛杉矶分校);Hong Kong University of Science and Technology(香港科技大学)

Yong Liu, Xiangning Chen, Minhao Cheng, Cho-Jui Hsieh, Yang You

摘要:

在使用大量GPU/TPU处理器训练神经网络时,大批量训练已成为一种常用技术。随着批量大小的增加,随机优化器往往收敛到尖锐的局部极小值,导致测试性能下降。目前的方法通常使用广泛的数据增强来增加批量大小,但我们发现数据增强的性能增益随着批量大小的增加而降低,并且数据增强在某个点之后变得不足。在本文中,我们建议使用对抗学习来增加大批量训练中的批量大小。尽管对抗学习是平滑决策表面和偏向平坦区域的自然选择,但它尚未成功应用于大批量训练,因为它需要在每一步至少进行两次连续的梯度计算,即使使用大量处理器,与普通训练相比,这至少会使运行时间增加一倍。为了克服这个问题,我们提出了一种新的并发对抗学习(ConAdv)方法,该方法通过利用静态参数来解耦对抗学习中的顺序梯度计算。实验结果表明,ConAdv可以成功地增加ImageNet上ResNet-50训练的批量大小,同时保持高精度。特别是,我们表明ConAdv在ImageNet ResNet-50训练中可以达到75.3\%的top-1准确率,批量大小为96 K,当将ConAdv与数据增强相结合时,准确率可以进一步提高到76.2\%。这是第一个成功将ResNet-50训练批量规模扩展到96 K的工作。

英文摘要:

Large-batch training has become a commonly used technique when training neural networks with a large number of GPU/TPU processors. As batch size increases, stochastic optimizers tend to converge to sharp local minima, leading to degraded test performance. Current methods usually use extensive data augmentation to increase the batch size, but we found the performance gain with data augmentation decreases as batch size increases, and data augmentation will become insufficient after certain point. In this paper, we propose to use adversarial learning to increase the batch size in large-batch training. Despite being a natural choice for smoothing the decision surface and biasing towards a flat region, adversarial learning has not been successfully applied in large-batch training since it requires at least two sequential gradient computations at each step, which will at least double the running time compared with vanilla training even with a large number of processors. To overcome this issue, we propose a novel Concurrent Adversarial Learning (ConAdv) method that decouple the sequential gradient computations in adversarial learning by utilizing staled parameters. Experimental results demonstrate that ConAdv can successfully increase the batch size on ResNet-50 training on ImageNet while maintaining high accuracy. In particular, we show ConAdv along can achieve 75.3\% top-1 accuracy on ImageNet ResNet-50 training with 96K batch size, and the accuracy can be further improved to 76.2\% when combining ConAdv with data augmentation. This is the first work successfully scales ResNet-50 training batch size to 96K.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑