基于Hessian的大批量训练及对抗鲁棒性分析
Hessian-based Analysis of Large Batch Training and Robustness to Adversaries
摘要:
神经网络的大批量训练已被证明在使用当前方法训练时会导致准确性损失。其确切的根本原因仍然没有完全弄清楚。在这里,我们通过Hessian算子和鲁棒优化来研究大批量训练。特别是,我们进行了一个基于Hessian的研究,以准确地分析在大批量训练时损失函数的景观如何变化。我们计算真正的海森谱,没有近似,通过反向传播的二阶导数。在多个网络上进行的大量实验表明,鞍点不是大批量训练泛化间隙的原因,并且结果一致表明,大批量收敛到具有明显更高Hessian谱的点。此外,我们表明,鲁棒训练允许人们喜欢平坦的区域,因为具有大Hessian谱的点对对抗性扰动的鲁棒性较差。我们进一步研究了这种关系,并提供了经验和理论证明,鲁棒训练的内环是一个无鞍优化问题\textit{almost everywhere}。我们提出了详细的实验与五种不同的网络架构,包括残余网络,测试MNIST,CIFAR-10和CIFAR-100数据集。我们已经开源了我们的方法,可以在[1]访问。
英文摘要:
Large batch size training of Neural Networks has been shown to incur accuracy loss when trained with the current methods. The exact underlying reasons for this are still not completely understood. Here, we study large batch size training through the lens of the Hessian operator and robust optimization. In particular, we perform a Hessian based study to analyze exactly how the landscape of the loss function changes when training with large batch size. We compute the true Hessian spectrum, without approximation, by back-propagating the second derivative. Extensive experiments on multiple networks show that saddle-points are not the cause for generalization gap of large batch size training, and the results consistently show that large batch converges to points with noticeably higher Hessian spectrum. Furthermore, we show that robust training allows one to favor flat areas, as points with large Hessian spectrum show poor robustness to adversarial perturbation. We further study this relationship, and provide empirical and theoretical proof that the inner loop for robust training is a saddle-free optimization problem \textit{almost everywhere}. We present detailed experiments with five different network architectures, including a residual network, tested on MNIST, CIFAR-10, and CIFAR-100 datasets. We have open sourced our method which can be accessed at [1].
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。