虚假特征如何被记忆:针对随机特征与NTK特征的精确分析
How Spurious Features Are Memorized: Precise Analysis for Random and NTK Features
AI总结:
该研究针对深度学习模型记忆训练数据中虚假特征的问题,通过模型样本稳定性与特征对齐度两项刻画记忆机制,在RF和NTK回归设置下推导精确理论结果,并在MNIST、CIFAR-10上验证了理论的预测能力。
摘要:
众所周知,深度学习模型会过拟合并记忆训练数据集中的虚假特征。尽管已有大量实证研究致力于理解这一现象,但目前仍缺乏一个可对其进行量化的严谨理论框架。本文中,我们研究与学习任务不相关的虚假特征,并通过两个独立项对其被记忆的机制给出精确刻画:(i)模型对单个训练样本的稳定性;(ii)虚假特征与完整样本之间的特征对齐度。第一项在学习理论中已有充分研究,且与经典研究中的泛化误差相关,而据我们所知,第二项是全新的。我们的核心技术成果针对随机特征(RF)和神经切线核(NTK)回归这两个典型设置,给出了特征对齐度的精确刻画。我们证明,虚假特征的记忆程度会随着泛化能力的提升而减弱,并且通过对特征对齐度的分析,我们揭示了模型及其激活函数的作用。数值实验表明,我们的理论在标准数据集(MNIST、CIFAR-10)上具备预测能力。
英文摘要:
Deep learning models are known to overfit and memorize spurious features in the training dataset. While numerous empirical studies have aimed at understanding this phenomenon, a rigorous theoretical framework to quantify it is still missing. In this paper, we consider spurious features that are uncorrelated with the learning task, and we provide a precise characterization of how they are memorized via two separate terms: (i) the stability of the model with respect to individual training samples, and (ii) the feature alignment between the spurious feature and the full sample. While the first term is well established in learning theory and it is connected to the generalization error in classical work, the second one is, to the best of our knowledge, novel. Our key technical result gives a precise characterization of the feature alignment for the two prototypical settings of random features (RF) and neural tangent kernel (NTK) regression. We prove that the memorization of spurious features weakens as the generalization capability increases and, through the analysis of the feature alignment, we unveil the role of the model and of its activation function. Numerical experiments show the predictive power of our theory on standard datasets (MNIST, CIFAR-10).
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。