不可腐蚀的神经网络：训练能够泛化到大内部扰动的模型
Incorruptible Neural Networks: Training Models that can Generalize to Large Internal Perturbations
AI总结：
本文提出通过SAM和RWP方法训练鲁棒于权重扰动的神经网络，理论和实验证明过度正则化的RWP在噪声鲁棒泛化中表现最优，同时动态调整扰动强度可提升优化效果。
摘要：
神经网络损失景观中的平坦区域长期以来被认为与更好的泛化特性相关。一个密切相关但不同的问题是训练能够抵御其权重内部扰动的模型，这可能是未来低功耗硬件平台的重要需求。在本文中，我们探索了使用两种方法，即尖锐度感知最小化（SAM）和随机权重扰动（RWP），以寻找对各种随机权重扰动具有鲁棒性的极小值。我们从两个角度考虑该问题：泛化（如何减少噪声鲁棒泛化差距）和优化（如何在强扰动下最大化优化器的性能）。首先，我们理论和实证地证明，过度正则化的RWP训练目标对于噪声鲁棒泛化是最优的。对于小幅度噪声，我们发现SAM的对抗性目标在任何RWP配置上进一步提高了性能，但在大幅度噪声下表现不佳。我们将其原因归因于损失景观不均匀性导致的梯度消失效应，影响了SAM和RWP。最后，我们展示了动态调整扰动强度以匹配损失景观的演变，可以提高这些扰动目标的优化。
英文摘要：
Flat regions of the neural network loss landscape have long been hypothesized to correlate with better generalization properties. A closely related but distinct problem is training models that are robust to internal perturbations to their weights, which may be an important need for future low-power hardware platforms. In this paper, we explore the usage of two methods, sharpness-aware minimization (SAM) and random-weight perturbation (RWP), to find minima robust to a variety of random corruptions to weights. We consider the problem from two angles: generalization (how do we reduce the noise-robust generalization gap) and optimization (how do we maximize performance from optimizers when subject to strong perturbations). First, we establish, both theoretically and empirically, that an over-regularized RWP training objective is optimal for noise-robust generalization. For small-magnitude noise, we find that SAM's adversarial objective further improves performance over any RWP configuration, but performs poorly for large-magnitude noise. We link the cause of this to a vanishing-gradient effect, caused by unevenness in the loss landscape, affecting both SAM and RWP. Lastly, we demonstrate that dynamically adjusting the perturbation strength to match the evolution of the loss landscape improves optimizing for these perturbed objectives.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
