Deep Neural Networks (DNNs) are prone to learning spurious features that correlate with the label during training but are irrelevant to the learning problem. This hurts model generalization and poses problems when deploying them in safety-critical applications. This paper aims to better understand the effects of spurious features through the lens of the learning dynamics of the internal neurons during the training process. We make the following observations: (1) While previous works highlight the harmful effects of spurious features on the generalization ability of DNNs, we emphasize that not all spurious features are harmful. Spurious features can be "benign" or "harmful" depending on whether they are "harder" or "easier" to learn than the core features for a given model. This definition is model and dataset-dependent. (2) We build upon this premise and use instance difficulty methods (like Prediction Depth (Baldock et al., 2021)) to quantify "easiness" for a given model and to identify this behavior during the training phase. (3) We empirically show that the harmful spurious features can be detected by observing the learning dynamics of the DNN's early layers. In other words, easy features learned by the initial layers of a DNN early during the training can (potentially) hurt model generalization. We verify our claims on medical and vision datasets, both simulated and real, and justify the empirical success of our hypothesis by showing the theoretical connections between Prediction Depth and information-theoretic concepts like V-usable information (Ethayarajh et al., 2021). Lastly, our experiments show that monitoring only accuracy during training (as is common in machine learning pipelines) is insufficient to detect spurious features. We, therefore, highlight the need for monitoring early training dynamics using suitable instance difficulty metrics.
Breaking Correlation Shift via Conditional Invariant Regularizer
University of Chinese Academy of Sciences(中国科学院大学);Academy of Mathematics and Systems Science, Chinese Academy of Sciences(中国科学院数学与系统科学研究院);Huawei Noah’s Ark Lab(华为诺亚方舟实验室)
When Does Group Invariant Learning Survive Spurious Correlations?
Academy of Mathematics and Systems Science, Chinese Academy of Sciences(中国科学院数学与系统科学研究院);University of Chinese Academy of Sciences(中国科学院大学);Baidu Inc.(百度公司);Institute for AI Industry Research, Tsinghua University(清华大学智能产业研究院);Beijing Academy of Artificial Intelligence(北京智源人工智能研究院)
Spurious Correlation Learning in Preference Optimization: Mechanisms, Consequences, and Mitigation via Tie Training
Department of Mathematics, Purdue University, West Lafayette IN, USA(普渡大学数学系);School of Mechanical Engineering, Purdue University, West Lafayette IN, USA(普渡大学机械工程学院);Massachusetts Institute of Technology, Cambridge MA, USA(麻省理工学院)
CausalAdv: Adversarial Robustness through the Lens of Causality
University of Science and Technology of China(中国科学技术大学);Hong Kong Baptist University(香港浸会大学);The University of Melbourne(墨尔本大学);The University of Sydney(悉尼大学);RIKEN Center for Advanced Intelligence Project(理化学研究所先进智能项目中心);Max Planck Institute for Intelligent Systems(马克斯·普朗克智能系统研究所);Carnegie Mellon University(卡内基梅隆大学);Mohamed bin Zayed University of Artificial Intelligence(穆罕默德·本·扎耶德人工智能大学)
University of Oxford, United Kingdom(牛津大学);Queen Mary University of London, United Kingdom(伦敦玛丽女王大学);Peking University, China(北京大学);University of Bristol, United Kingdom(布里斯托大学);King's College London, United Kingdom(伦敦国王学院);Hong Kong University of Science(香港科学大学);University College London, United Kingdom(伦敦大学学院)
The Clever Hans Mirage: A Comprehensive Survey on Spurious Correlations in Machine Learning
University of Virginia(弗吉尼亚大学);Purdue University(普渡大学);University of Illinois Urbana-Champaign(伊利诺伊大学厄巴纳-香槟分校);Pennsylvania State University(宾夕法尼亚州立大学);Amazon(亚马逊);New York University(纽约大学)