白化提高线性探针对虚假相关性的鲁棒性
Whitening Improves Robustness to Spurious Correlations in Linear Probes
浏览论文内容
中文总结 AI 辅助
本研究提出白化预处理，通过均衡协方差矩阵特征值，减少线性探针对虚假相关性的依赖，从而提升模型鲁棒性，并在合成数据及标准基准上验证了其有效性。
中文摘要 AI 辅助
深度神经网络倾向于依赖可能具有虚假性的简单特征，从而难以泛化。我们在线性探针的设置中研究这一问题，其中（广义）线性模型被拟合到（预训练）模型的表示上。我们利用这些模型与最大间隔分类器的联系，表明它们偏好与协方差矩阵的大特征值相关的方向。白化通过均衡协方差矩阵的特征值来消除这种偏好。这一观察促使白化成为一种预处理步骤，可以在不需要事先知道虚假相关性存在或不需要标注数据的情况下，减少对虚假相关性的依赖。我们检查了白化对合成数据生成过程和标准虚假相关性基准的影响，发现它提高了鲁棒性。我们还发现，将白化添加到现有方法中可以提高鲁棒性。
英文摘要
Deep neural networks tend to rely on simple features that may be spurious and thus fail to generalize. We study this problem in the setting of linear probes, where a (generalized) linear model is fitted on the representations of a (pretrained) model. We use the connection of these models to the max-margin classifier, and show they favor directions associated with large eigenvalues of the covariance matrix. Whitening removes this preference by equalizing the eigenvalues of the covariance matrix. This observation motivates whitening as a preprocessing step that can reduce reliance on spurious correlations without requiring prior knowledge of their presence or labeled data. We examine the effect of whitening on a synthetic data-generating process and standard spurious correlation benchmarks, and find that it improves robustness. We also find that whitening can improve robustness when added to existing approaches.
发表机构
- University of Amsterdam（阿姆斯特丹大学）
- Tinbergen Institute（丁伯根研究所）
机构由 AI 辅助整理，请以论文原文为准。
