arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2602.00130cs.LG
收藏

关于深度神经网络中表示几何与泛化能力之间关系的研究

On the Relationship Between Representation Geometry and Generalization in Deep Neural Networks

Institute of Engineering, Pulchowk Campus, Tribhuvan University(特里布文大学普尔乔克校区工程学院)

Sumit Yadav

AI总结:

研究发现有效维度能预测深度神经网络的性能,且与模型容量无关,通过噪声实验验证了其因果关系。

摘要:

我们研究了表示几何与神经网络性能之间的关系。分析了52个预训练的ImageNet模型,涵盖了13种架构家族,我们发现有效维度——一个无监督的几何度量——强烈预测了准确性。输出有效维度在考虑模型容量后,达到部分相关性r=0.75(p<10^(-10)),而总压缩度量达到部分相关性r=-0.72。这些发现在ImageNet和CIFAR-10上得到复制,并扩展到NLP:有效维度预测了8个编码器模型在SST-2/MNLI上的性能以及15个解码器-only LLMs在AG News上的性能(r=0.69,p=0.004),而模型大小则没有(r=0.07)。我们确立了双向因果关系:通过噪声降级几何结构会导致准确性下降(r=-0.94,p<10^(-9)),而通过PCA改进几何结构可以在不同架构中保持准确性(在95%方差下减少0.03pp)。这种关系对噪声类型不敏感——高斯、均匀、丢弃和盐和胡椒噪声都显示|r|>0.90。这些结果表明,有效维度提供了关于神经网络性能的领域无关的预测和因果信息,完全无需标签。

英文摘要:

We investigate the relationship between representation geometry and neural network performance. Analyzing 52 pretrained ImageNet models across 13 architecture families, we show that effective dimension -- an unsupervised geometric metric -- strongly predicts accuracy. Output effective dimension achieves partial r=0.75 ($p < 10^(-10)$) after controlling for model capacity, while total compression achieves partial r=-0.72. These findings replicate across ImageNet and CIFAR-10, and generalize to NLP: effective dimension predicts performance for 8 encoder models on SST-2/MNLI and 15 decoder-only LLMs on AG News (r=0.69, p=0.004), while model size does not (r=0.07). We establish bidirectional causality: degrading geometry via noise causes accuracy loss (r=-0.94, $p < 10^(-9)$), while improving geometry via PCA maintains accuracy across architectures (-0.03pp at 95% variance). This relationship is noise-type agnostic -- Gaussian, Uniform, Dropout, and Salt-and-pepper noise all show $|r| > 0.90$. These results establish that effective dimension provides domain-agnostic predictive and causal information about neural network performance, computed entirely without labels.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑