正交梯度约束塑造噪声标签记忆动态
Orthogonal Gradient Constraints Shape Noisy-Label Memorization Dynamics
浏览论文内容
中文总结 AI 辅助
研究噪声标签学习中记忆驱动过拟合问题，核心方法是用OrthoGrad去除权重梯度中与当前权重向量平行的分量，在MNIST小数据及CIFAR-10 ResNet-18实验中验证其能改变记忆轨迹，但依赖数据情况，可作学习动态诊断。
中文摘要 AI 辅助
现代神经网络能够拟合错误的训练标签，使得噪声标签学习成为研究记忆驱动过拟合的有用场景。大多数正则化方法修改目标、架构或数据分布；本文转而研究对优化器更新本身的几何干预。我们评估了OrthoGrad，它去除每个权重梯度中与当前权重向量平行的分量，用于噪声标签图像分类。在小数据情况下的MNIST上，OrthoGrad对卷积神经网络最明显地提高了测试准确率，同时减少了对错误标签的拟合。基于权重范数和梯度-权重余弦相似度的机制诊断表明，当原始梯度包含非平凡径向分量时，投影效果最强，而在大数据情况下梯度已几乎与权重正交时效果减弱。CIFAR-10 ResNet-18的额外实验表明，该方法可以改变记忆轨迹，但不能防止最终的噪声标签记忆。这些结果支持正交更新约束作为研究学习动态的有用诊断，同时表明OrthoGrad依赖于数据情况，而非普遍正则化。
英文摘要
Modern neural networks can fit corrupted training labels, making noisy-label learning a useful setting for studying memorization-driven overfitting. Most regularization methods modify the objective, architecture, or data distribution; here we instead study a geometric intervention on the optimizer update itself. We evaluate OrthoGrad, which removes the component of each weight gradient parallel to the current weight vector, in noisy-label image classification. On MNIST with small-data regimes, OrthoGrad improves test accuracy most clearly for CNNs while reducing corrupted-label fitting. Mechanism diagnostics based on weight norms and gradient-weight cosine similarity suggest that the projection has the strongest effect when the raw gradient contains a nontrivial radial component, and becomes weaker in larger-data regimes where gradients are already nearly orthogonal to weights. Additional CIFAR-10 ResNet-18 experiments show that the method can alter memorization trajectories but does not prevent eventual noisy-label memorization. These results support orthogonal update constraints as a useful diagnostic for studying learning dynamics, while showing that OrthoGrad is regime-dependent rather than universally regularizing.
