在固定置信度分布下可靠性漂移能有多大？
How Much Can Reliability Drift Under a Fixed Confidence Distribution?
- Adelaide University（阿德莱德大学）
机构由 AI 辅助整理，请以论文原文为准。
AI总结：
本文研究在置信度分布固定时分类器可靠性漂移的最坏情况，提出脆弱性剖面方法，并在ImageNet上验证了其有效性。
AI中文摘要：
分类器的条件准确率可能在其置信度分布完全保持不变的情况下发生变化。我们研究了在保持置信度分数分布的协变量偏移下，可靠性关系的最坏情况移动，通过在每个置信度水平内施加 χ² 预算来约束重新加权；由此产生的最坏情况，作为预算的函数，即为脆弱性剖面。在可从源分布计算的预算区间内，该剖面恰好等于预算的平方根乘以正确性倾向的层内方差——即校准-细化分解中的分组损失项。超出此区间，剖面由倾向定律的尾部控制，整个向上剖面决定了居中的层内定律；因此，校准残差和分组方差通常不能决定脆弱性，尽管在标签和预测是确定性的情况下它们可以。由于倾向不可观测，我们将重新加权限制在置信度区间内的学习有限读出，通过读出单元内剩余的分组方差来界定限制所遗漏的部分，使用角色分离的标签估计受限剖面，并提供单独的分割样本下置信界。在ImageNet上，该界在六个主要分类器中的四个和十二个额外分类器中的九个（按发布版本）的两个分割中均为正，在温度缩放后十八个中的三个为正。在未使用评估标签拟合的优化重新加权下，保持的漂移跟踪估计剖面；探索性标签置换诊断对该统计量产生接近零的一致性，同时大体上重现了无符号随机重新加权所观察到的相关性。
英文摘要：
A classifier's conditional accuracy can change while its confidence distribution stays exactly the same. We study the worst-case movement of the reliability relation under covariate shifts that preserve the distribution of the confidence score, constraining the reweighting within each confidence level by a $χ^2$ budget; the resulting worst case, as a function of the budget, is a fragility profile. On an interval of budgets that can be computed from the source distribution, the profile equals exactly the square root of the budget times the within-level variance of the correctness propensity -- the grouping-loss term of calibration-refinement decompositions. Beyond this interval the profile is governed by the tails of the propensity law, and the entire upward profile determines the centred within-level law; consequently, calibration residual and grouping variance do not determine fragility in general, though they do when labels and predictions are deterministic. Since the propensity is not observed, we restrict reweightings to a learned finite readout within confidence bins, bound the part the restriction misses by the grouping variance remaining inside readout cells, estimate the restricted profile with role-separated labels, and provide a separate split-sample lower confidence bound. On ImageNet this bound is positive in both splits for four of six primary classifiers and nine of twelve additional ones as released, and for three of eighteen after temperature scaling. Held-out drift under optimised reweightings fitted without evaluation labels tracks the estimated profile; an exploratory label-permutation diagnostic yields near-zero agreement for this statistic while largely reproducing the correlation observed for unsigned random reweightings.
