arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2603.02293cs.LGcs.AI
收藏

恶性尾部:过参数化网络中标签噪声的谱分离

Spectral Overfitting in Noisy Linear Probing of Pretrained Representations

Zice Wang, Zhenyu Zhang

AI总结:

研究揭示过参数化网络中标签噪声的恶性尾部现象,通过谱线性探针发现SGD无法抑制噪声,反而将其推向高频正交子空间,提出几何截断方法以提升泛化能力。

摘要:

尽管隐式正则化在低噪声环境下促进良性过拟合,近期理论工作预测噪声信号比增加时会出现有害过拟合的相变。我们通过实验隔离这一转变的几何机制:恶性尾部是一种失败模式,其中网络功能上分离信号和噪声,将相干语义特征压缩到低秩子空间,同时将随机标签噪声推入高频正交成分,与系统性或损坏对齐的噪声不同。通过训练动态的谱线性探针,我们证明SGD未能抑制这种噪声,反而隐式地将其偏向高频正交子空间,从而有效保持信号-噪声分离性。我们展示这种几何分离不同于未训练模型中的简单方差减少。在训练网络中,SGD主动分离噪声,允许事后显式谱截断(d << D)来手术性修剪噪声主导的子空间。这种方法恢复了收敛模型中潜在的最佳泛化能力。与不稳定的时间早停不同,几何截断提供了一种稳定的后处理干预。我们的发现表明,在标签噪声下,超额谱容量并非无害的冗余,而是一种潜在的结构隐患,允许噪声记忆,需要显式秩约束来过滤随机损坏以实现稳健泛化。

英文摘要:

Frozen pretrained features are often treated as a safe interface for downstream learning: only a small linear readout is trained, while the backbone is fixed. We show that this readout can still overfit noisy labels in a structured way. A label-blind PCA rank sweep reveals a sharp spectral pattern: under label noise, exposing all pretrained directions can hurt clean accuracy, and intermediate ranks often recover much of the lost performance. Rank-matched random projections help less, and measured between-class signal is strongly concentrated in leading PCs. The pattern appears across three ImageNet-pretrained backbones on CIFAR-10, with gains up to $36.0\pm0.8$ points over the default full-rank probe at 40\% noise. Tuned full-rank probes outperform validation-selected PCA probes, so we present the sweep as a diagnostic of spectral overfitting rather than a competitive noisy-label method.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑