诊断分布偏移下的模型性能
Diagnosing Model Performance Under Distribution Shift
AI总结：
提出 DISDE 方法，将分布偏移导致的模型性能下降分解为样本难度、特征-结果关系及未见样本三类因素，并用重加权估计假设分布性能。
摘要：
当预测模型部署到与训练分布不同的目标分布时，其表现可能很差。为理解这些运行中的失效模式，我们开发了一种名为 DIstribution Shift DEcomposition（DISDE，分布偏移分解）的方法，将性能下降归因于不同类型的分布偏移。我们的方法把性能下降分解为三个部分：1）训练中较难但常见样本的增加；2）特征与结果之间关系的变化；3）在训练期间少见或未见样本上的较差表现。这些项通过两种方式定义：固定 X 上的分布，同时在训练与目标之间改变 Y | X 的条件分布；或固定 Y | X 的条件分布，同时改变 X 上的分布。为此，我们定义了一个假设的 X 分布，由训练和目标中共同常见的取值组成；在该分布上很容易比较 Y | X，从而比较预测性能。我们通过重加权方法估计这一假设分布上的性能。在实证中，我们展示了该方法如何：1）为表格人口普查数据上的就业预测在分布偏移下提供潜在建模改进的信息；2）帮助解释为什么某些领域自适应方法未能提升卫星图像分类的模型性能。
英文摘要：
Prediction models can perform poorly when deployed to target distributions different from the training distribution. To understand these operational failure modes, we develop a method, called DIstribution Shift DEcomposition (DISDE), to attribute a drop in performance to different types of distribution shifts. Our approach decomposes the performance drop into terms for 1) an increase in harder but frequently seen examples from training, 2) changes in the relationship between features and outcomes, and 3) poor performance on examples infrequent or unseen during training. These terms are defined by fixing a distribution on $X$ while varying the conditional distribution of $Y \mid X$ between training and target, or by fixing the conditional distribution of $Y \mid X$ while varying the distribution on $X$. In order to do this, we define a hypothetical distribution on $X$ consisting of values common in both training and target, over which it is easy to compare $Y \mid X$ and thus predictive performance. We estimate performance on this hypothetical distribution via reweighting methods. Empirically, we show how our method can 1) inform potential modeling improvements across distribution shifts for employment prediction on tabular census data, and 2) help to explain why certain domain adaptation methods fail to improve model performance for satellite image classification.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
