FixNorm：剖析训练深度神经网络的权重衰减
FixNorm: Dissecting Weight Decay for Training Deep Neural Networks
摘要：
权重衰减是一种广泛用于训练深度神经网络（DNN）的技术。它极大地影响泛化性能，但其潜在机制尚未完全了解。最近的研究表明，对于层之后的归一化，权重衰减主要影响有效的学习率。然而，尽管规范化在现代DNN中被广泛采用，但最终的全连接层等层并不满足这一前提条件。对于这些层，权重衰减的影响仍然不清楚。本文全面研究了权值衰减的机制，发现除了影响有效学习率外，权重衰减还有另一个同样重要的机制：通过控制跨界风险来影响泛化性能。这两种机制一起对权重衰减的影响给出了更全面的解释。基于这一发现，我们提出了一种新的训练方法，称为FixNorm，它摒弃了权重衰减，直接控制这两种机制。我们还提出了一种简单而有效的方法来调整FixNorm的超参数，它可以在几次试验中找到接近最优的解决方案。在ImageNet分类任务中，使用FixNorm训练的EfficientNet-B 0达到了77.7%，明显优于原始基线。令人惊讶的是，当将MobileNetV 2扩展到相同的FLOPS并使用EfficientNet-B 0应用相同的技巧时，使用FixNorm进行训练的效果达到77.4%，仅低0.3%。一系列的SOTA结果表明，良好的调整培训程序的重要性，并进一步验证了我们的方法的有效性。我们使用FixNorm设置了更好的基线，以促进社区中的公平比较。
英文摘要：
Weight decay is a widely used technique for training Deep Neural Networks(DNN). It greatly affects generalization performance but the underlying mechanisms are not fully understood. Recent works show that for layers followed by normalizations, weight decay mainly affects the effective learning rate. However, despite normalizations have been extensively adopted in modern DNNs, layers such as the final fully-connected layer do not satisfy this precondition. For these layers, the effects of weight decay are still unclear. In this paper, we comprehensively investigate the mechanisms of weight decay and find that except for influencing effective learning rate, weight decay has another distinct mechanism that is equally important: affecting generalization performance by controlling cross-boundary risk. These two mechanisms together give a more comprehensive explanation for the effects of weight decay. Based on this discovery, we propose a new training method called FixNorm, which discards weight decay and directly controls the two mechanisms. We also propose a simple yet effective method to tune hyperparameters of FixNorm, which can find near-optimal solutions in a few trials. On ImageNet classification task, training EfficientNet-B0 with FixNorm achieves 77.7%, which outperforms the original baseline by a clear margin. Surprisingly, when scaling MobileNetV2 to the same FLOPS and applying the same tricks with EfficientNet-B0, training with FixNorm achieves 77.4%, which is only 0.3% lower. A series of SOTA results show the importance of well-tuned training procedures, and further verify the effectiveness of our approach. We set up more well-tuned baselines using FixNorm, to facilitate fair comparisons in the community.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
