arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2401.08876cs.HCcs.CVcs.LG
收藏

评估共形预测集在AI辅助图像标注中的实用性

Evaluating the Utility of Conformal Prediction Sets for AI-Advised Image Labeling

Northwestern University(西北大学)

Dongping Zhang, Angelos Chatzimparmpas, Negar Kamali, Jessica Hullman

AI总结:

研究通过在线实验比较共形预测集与Top-1/Top-k显示在AI辅助图像标注中的实用性,发现预测集在标注分布外图像时具一定优势,尤其当集合较小时。

摘要:

随着深度神经网络越来越普遍地部署在高风险领域,其黑盒性质使得不确定性量化充满挑战。我们研究了共形预测集的呈现方式——这是一种无分布的生成具有指定覆盖率的预测集的方法类别——以在AI辅助决策中表达不确定性。通过一项大型在线实验,我们比较了共形预测集与Top-1和Top-k预测显示在AI辅助图像标注中的实用性。在预注册分析中,我们发现预测集对准确性的实用性随任务难度而变化:对于简单图像,其准确率与Top-1和Top-k显示相当或更低;但在我们研究的场景中,当集合大小较小时,预测集在协助人类标注分布外(OOD)图像方面具有一定优势。我们的结果实证地指出了共形预测集的实际挑战,并为其如何融入现实世界决策提供了启示。

英文摘要:

As deep neural networks are more commonly deployed in high-stakes domains, their black-box nature makes uncertainty quantification challenging. We investigate the presentation of conformal prediction sets--a distribution-free class of methods for generating prediction sets with specified coverage--to express uncertainty in AI-advised decision-making. Through a large online experiment, we compare the utility of conformal prediction sets to displays of Top-1 and Top-k predictions for AI-advised image labeling. In a pre-registered analysis, we find that the utility of prediction sets for accuracy varies with the difficulty of the task: while they result in accuracy on par with or less than Top-1 and Top-k displays for easy images, prediction sets offer some advantage in assisting humans in labeling out-of-distribution (OOD) images in the setting that we studied, especially when the set size is small. Our results empirically pinpoint practical challenges of conformal prediction sets and provide implications on how to incorporate them for real-world decision-making.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑