领域特征坍塌：对分布外检测的影响及解决方案
Domain Feature Collapse: Implications for Out-of-Distribution Detection and Solutions
AI总结：
本文通过信息论解释了单一领域训练导致分布外检测失败的机制，提出领域特征坍塌是根本原因，并通过领域过滤方法验证了理论，为迁移学习和模型微调提供了新视角。
摘要：
为什么最先进的分布外检测方法在模型在单一领域数据集上训练时会出现灾难性失败？我们通过信息论的视角提供了这种现象的第一个理论解释。我们证明了在单一领域数据上进行监督学习不可避免地会产生领域特征坍塌——即表示中I(x_d; z)=0，意味着完全丢弃了领域特定信息。这是信息瓶颈优化的根本结果：模型在单一领域（例如医学图像）上训练时，只学习依赖类别特定特征而丢弃领域特征，导致在检测分布外样本时（例如在MNIST上仅达到53%的FPR@95）出现灾难性失败。我们通过Fano不等式扩展了我们的分析，以量化实际场景中的部分坍塌。为了验证我们的理论，我们引入了Domain Bench，一个单一领域数据集的基准，并展示了通过领域过滤（使用预训练表示）保留I(x_d; z) > 0可以解决失败模式。虽然领域过滤本身在概念上很简单，但其有效性为我们的信息论框架提供了强有力的实证证据。我们的工作解释了一个令人困惑的实证现象，揭示了监督学习在狭窄领域中的根本限制，并对迁移学习以及何时微调或冻结预训练模型有更广泛的影响。
英文摘要：
Why do state-of-the-art OOD detection methods exhibit catastrophic failure when models are trained on single-domain datasets? We provide the first theoretical explanation for this phenomenon through the lens of information theory. We prove that supervised learning on single-domain data inevitably produces domain feature collapse -- representations where I(x_d; z) = 0, meaning domain-specific information is completely discarded. This is a fundamental consequence of information bottleneck optimization: models trained on single domains (e.g., medical images) learn to rely solely on class-specific features while discarding domain features, leading to catastrophic failure when detecting out-of-domain samples (e.g., achieving only 53% FPR@95 on MNIST). We extend our analysis using Fano's inequality to quantify partial collapse in practical scenarios. To validate our theory, we introduce Domain Bench, a benchmark of single-domain datasets, and demonstrate that preserving I(x_d; z) > 0 through domain filtering (using pretrained representations) resolves the failure mode. While domain filtering itself is conceptually straightforward, its effectiveness provides strong empirical evidence for our information-theoretic framework. Our work explains a puzzling empirical phenomenon, reveals fundamental limitations of supervised learning in narrow domains, and has broader implications for transfer learning and when to fine-tune versus freeze pretrained models.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
