当人类不同意时停止测量校准
Stop Measuring Calibration When Humans Disagree
摘要:
校准是一种流行的框架,用于评估分类器是否知道它何时不知道-即,其预测概率是预测正确的可能性的良好指示。正确性通常是针对人类多数阶级来估计的。最近,对人类多数的校准已经在人类固有地不同意哪个类适用的任务上进行了测量。我们表明,考虑到固有的分歧,测量校准到人类大多数在理论上是有问题的,在ChaosNLI数据集上经验性地证明了这一点,并推导出几个实例级的校准措施,这些措施捕获了人类判断的关键统计特性-类频率,排名和熵。
英文摘要:
Calibration is a popular framework to evaluate whether a classifier knows when it does not know - i.e., its predictive probabilities are a good indication of how likely a prediction is to be correct. Correctness is commonly estimated against the human majority class. Recently, calibration to human majority has been measured on tasks where humans inherently disagree about which class applies. We show that measuring calibration to human majority given inherent disagreements is theoretically problematic, demonstrate this empirically on the ChaosNLI dataset, and derive several instance-level measures of calibration that capture key statistical properties of human judgements - class frequency, ranking and entropy.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。