arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2307.09302cs.LGcs.CVstat.MEstat.ML
收藏

模糊真实标签下的保形预测

Conformal prediction under ambiguous ground truth

Google DeepMind(谷歌DeepMind);Google(谷歌)

David Stutz, Abhijit Guha Roy, Tatiana Matejovicova, Patricia Strachan, Ali Taylan Cemgil, Arnaud Doucet

AI总结:

针对专家标注存在分歧时传统保形预测覆盖率不足的问题,提出利用专家意见构建非退化分布并开发Monte Carlo CP方法,通过采样合成伪标签提供理论保证,有效弥补了覆盖差距。

摘要:

保形预测(Conformal Prediction, CP)通过构建满足 $\mathbb{P}(Y \in C(X))\geq 1-\alpha$ 的预测集 $C(X)$ 来进行严格的不确定性量化,其中 $\alpha\in [0,1]$ 由用户指定,并依赖于来自 $\mathbb{P}=\mathbb{P}^{X} \otimes \mathbb{P}^{Y|X}$ 的校准数据 $(X_1,Y_1),...,(X_n,Y_n)$。通常隐式假设 $\mathbb{P}^{Y|X}$ 是“真实”的后验标签分布。然而,在许多现实场景中,标签 $Y_1,...,Y_n$ 是通过投票程序聚合专家意见获得的,从而产生 one-hot 分布 $\mathbb{P}_{vote}^{Y|X}$。对于此类“投票”标签,CP 的保证因此是针对 $\mathbb{P}_{vote}=\mathbb{P}^X \otimes \mathbb{P}_{vote}^{Y|X}$ 而非真实分布 $\mathbb{P}$。在真实标签无歧义的情况下,$\mathbb{P}_{vote}$ 和 $\mathbb{P}$ 之间的区别无关紧要。然而,当专家因标签模糊而无法达成一致时,用 one-hot 分布 $\mathbb{P}_{vote}^{Y|X}$ 近似 $\mathbb{P}^{Y|X}$ 会忽略这种不确定性。在本文中,我们提出利用专家意见,使用非退化分布 $\mathbb{P}_{agg}^{Y|X}$ 来近似 $\mathbb{P}^{Y|X}$。我们开发了 Monte Carlo CP 程序,通过从 $\mathbb{P}_{agg}^{Y|X}$ 中为每个校准样本 $X_1,...,X_n$ 采样多个合成伪标签,提供针对 $\mathbb{P}_{agg}=\mathbb{P}^X \otimes \mathbb{P}_{agg}^{Y|X}$ 的保证。在专家标注者之间存在显著分歧的皮肤状况分类案例研究中,我们表明针对 $\mathbb{P}_{vote}$ 应用 CP 会导致对专家标注的覆盖率不足:在设定为 $72\\%$ 覆盖率进行校准时,平均短缺 $10\\%$;我们的 Monte Carlo CP 在经验和理论上都弥补了这一差距。

英文摘要:

Conformal Prediction (CP) allows to perform rigorous uncertainty quantification by constructing a prediction set $C(X)$ satisfying $\mathbb{P}(Y \in C(X))\geq 1-α$ for a user-chosen $α\in [0,1]$ by relying on calibration data $(X_1,Y_1),...,(X_n,Y_n)$ from $\mathbb{P}=\mathbb{P}^{X} \otimes \mathbb{P}^{Y|X}$. It is typically implicitly assumed that $\mathbb{P}^{Y|X}$ is the "true" posterior label distribution. However, in many real-world scenarios, the labels $Y_1,...,Y_n$ are obtained by aggregating expert opinions using a voting procedure, resulting in a one-hot distribution $\mathbb{P}_{vote}^{Y|X}$. For such ``voted'' labels, CP guarantees are thus w.r.t. $\mathbb{P}_{vote}=\mathbb{P}^X \otimes \mathbb{P}_{vote}^{Y|X}$ rather than the true distribution $\mathbb{P}$. In cases with unambiguous ground truth labels, the distinction between $\mathbb{P}_{vote}$ and $\mathbb{P}$ is irrelevant. However, when experts do not agree because of ambiguous labels, approximating $\mathbb{P}^{Y|X}$ with a one-hot distribution $\mathbb{P}_{vote}^{Y|X}$ ignores this uncertainty. In this paper, we propose to leverage expert opinions to approximate $\mathbb{P}^{Y|X}$ using a non-degenerate distribution $\mathbb{P}_{agg}^{Y|X}$. We develop Monte Carlo CP procedures which provide guarantees w.r.t. $\mathbb{P}_{agg}=\mathbb{P}^X \otimes \mathbb{P}_{agg}^{Y|X}$ by sampling multiple synthetic pseudo-labels from $\mathbb{P}_{agg}^{Y|X}$ for each calibration example $X_1,...,X_n$. In a case study of skin condition classification with significant disagreement among expert annotators, we show that applying CP w.r.t. $\mathbb{P}_{vote}$ under-covers expert annotations: calibrated for $72\%$ coverage, it falls short by on average $10\%$; our Monte Carlo CP closes this gap both empirically and theoretically.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑