过度参数化加剧伪相关的原因研究
An Investigation of Why Overparameterization Exacerbates Spurious Correlations
摘要：
我们研究了为什么过度参数化-增加模型大小远远超过零训练误差点-可以伤害少数群体的测试误差，尽管提高了平均测试误差时，有虚假的相关性的数据。通过对两个图像数据集的模拟和实验，我们确定了驱动这种行为的训练数据的两个关键属性：多数与少数群体的比例，以及虚假相关性的信噪比。然后，我们分析了一个线性设置，并从理论上展示了如何对“记忆”更少的例子模型的归纳偏见可能会导致overparameterization伤害。我们的分析导致了一种违反直觉的方法，即对多数群体进行二次抽样，这种方法在过度参数化的制度中根据经验实现了较低的少数群体误差，即使增加少数群体权重的标准方法失败了。总的来说，我们的研究结果表明，使用过参数化模型与使用所有训练数据以实现低最差组误差之间存在紧张关系。
英文摘要：
We study why overparameterization -- increasing model size well beyond the point of zero training error -- can hurt test error on minority groups despite improving average test error when there are spurious correlations in the data. Through simulations and experiments on two image datasets, we identify two key properties of the training data that drive this behavior: the proportions of majority versus minority groups, and the signal-to-noise ratio of the spurious correlations. We then analyze a linear setting and theoretically show how the inductive bias of models towards "memorizing" fewer examples can cause overparameterization to hurt. Our analysis leads to a counterintuitive approach of subsampling the majority group, which empirically achieves low minority error in the overparameterized regime, even though the standard approach of upweighting the minority fails. Overall, our results suggest a tension between using overparameterized models versus using all the training data for achieving low worst-group error.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
