学习可逆输出映射可以减轻神经网络中的简单性偏差
Learning an Invertible Output Mapping Can Mitigate Simplicity Bias in Neural Networks
摘要：
众所周知，深度神经网络与训练分布相比，即使是微小的分布偏移也是脆弱的。虽然一项工作已经证明DNN的简单性偏差（SB）-偏向于只学习最简单的特征-是这种脆弱性的关键原因，但最近的另一项工作令人惊讶地发现，主干确实学习了多样/复杂的特征，它们的脆弱性是由于线性分类头主要依赖于最简单的特征。为了弥合这两条工作线之间的差距，我们首先假设并验证，虽然SB可能不会完全排除学习复杂的功能，它放大了复杂的功能。也就是说，简单的特征在学习的表征中被复制多次，而复杂的特征可能不会被复制。这种现象，我们称之为特征复制假设，再加上SGD的隐式偏差收敛到特征空间中的最大边缘解，导致模型主要依赖于简单的特征进行分类。为了减轻这种偏差，我们提出了特征重建正则化器（FRR），以确保学习的特征可以从logits中重建回来。在线性层训练（FRR-L）中使用{ \em FRR}鼓励使用更多样化的特征进行分类。我们进一步建议通过冻结使用FRR-L训练的线性层的权重来微调整个网络，以细化学习的特征，使其更适合分类。使用这个简单的解决方案，我们证明了高达15%的收益，在最近推出的半合成数据集的极端分布偏移的OOD精度。此外，我们还在标准OOD基准DomainBed上展示了现有SOTA方法的显著收益。
英文摘要：
Deep Neural Networks are known to be brittle to even minor distribution shifts compared to the training distribution. While one line of work has demonstrated that Simplicity Bias (SB) of DNNs - bias towards learning only the simplest features - is a key reason for this brittleness, another recent line of work has surprisingly found that diverse/ complex features are indeed learned by the backbone, and their brittleness is due to the linear classification head relying primarily on the simplest features. To bridge the gap between these two lines of work, we first hypothesize and verify that while SB may not altogether preclude learning complex features, it amplifies simpler features over complex ones. Namely, simple features are replicated several times in the learned representations while complex features might not be replicated. This phenomenon, we term Feature Replication Hypothesis, coupled with the Implicit Bias of SGD to converge to maximum margin solutions in the feature space, leads the models to rely mostly on the simple features for classification. To mitigate this bias, we propose Feature Reconstruction Regularizer (FRR) to ensure that the learned features can be reconstructed back from the logits. The use of {\em FRR} in linear layer training (FRR-L) encourages the use of more diverse features for classification. We further propose to finetune the full network by freezing the weights of the linear layer trained using FRR-L, to refine the learned features, making them more suitable for classification. Using this simple solution, we demonstrate up to 15% gains in OOD accuracy on the recently introduced semi-synthetic datasets with extreme distribution shifts. Moreover, we demonstrate noteworthy gains over existing SOTA methods on the standard OOD benchmark DomainBed as well.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
