发表机构
Princeton University（普林斯顿大学）
机构由 AI 辅助整理，请以论文原文为准。AI 中文总结
研究探索交错噪声注入在优化中的效果，通过理论分析揭示其作用机制，引入梯度范数稳定技术，与其他方法比较发现能显著提升在CIFAR-100-C等数据集上的抗噪及分布外数据性能，是提升测试性能的有效工具。
AI 中文摘要
噪声注入是随机优化中的一项知名技术。本文报告了其在采用交错（开-关-开-关……）而非通常单调衰减调度时的惊人有效性。对噪声注入进行了理论分析，证实脉冲噪声的干扰近似于雅可比正则化，高斯噪声起到曲率惩罚作用。交错调度能产生更好结果，为稳定训练，引入基于干净梯度幅度缩放噪声更新的梯度范数稳定技术。与其他常见增强方法比较，在多种数据集和架构上有显著改进，通过显著性和注意力图表明其作用机制，交错噪声注入能有效提升各类数据测试性能且计算成本基本为零。
英文摘要
Noise injection is a well-known technique in stochastic optimization. We report its surprising effectiveness with an interleaved (on-off-on-off...) rather than the usual monotonic decay schedule. We present a theoretical analysis of noise injection, which confirms that corruption by impulse noise approximates a Jacobian regularization, whereas Gaussian noise acts as a curvature penalty. This regularization behavior has been invoked to explain why noise injection increases model robustness. But the interleaved nature of our proposed schedule produces superior results even for the optimization objective: mixing phases of noisy data permits the optimizer to escape local minima and increase exploration without the risk of catastrophically forgetting the important features from the clean data. To stabilize this training scheme against the rapid changes of the loss when switching between clean and noisy data, we introduce a gradient-norm stabilization technique that scales noisy updates based on clean gradient magnitudes. We compare this method with other common augmentation methods and find substantial improvements in corruption tolerance and robustness to real-world distribution shifts on CIFAR-100-C, ImageNet-C, and ImageNet-R for ResNet and ViT architectures, with the best results being achieved by stacking our method on top of other augmentations. Through saliency and attention maps we show that the effect of interleaved noise injection stems from penalizing the failure modes encouraged by the inductive bias of the models: impulse noise works against the locality bias of convolutional (ResNet) architectures, and Gaussian noise reduces the tendency of attention-based models to pick up large-scale spurious features. Interleaved noise injection is therefore an effective tool to improve the test performance on clean, noisy, and out-of-distribution data at essentially zero computational cost.
Comments24 pages, 11 figures
