arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2205.01411cs.AIcs.HC
收藏

预测集在人工智能团队中的效用

On the Utility of Prediction Sets in Human-AI Teams

University of Cambridge(剑桥大学);The Alan Turing Institute(艾伦·图灵研究所)

Varun Babbar, Umang Bhatt, Adrian Weller

摘要:

对人工智能团队的研究通常为专家提供单一标签,这忽略了模型推荐中的不确定性。共形预测(CP)是一个成熟的研究路线,专注于构建一个理论上有基础的,校准的预测集,其中可能包含多个标签。我们将探讨这种预测集如何影响人类AI团队中的专家决策。我们对人类受试者的评估发现,集值预测对专家有积极影响。然而,我们注意到CP提供的预测集可能非常大,这导致了无用的AI助手。为了减轻这种情况,我们引入了D-CP,这是一种在一些示例上执行CP并尊重专家的方法。我们证明了D-CP可以减少非延迟样本的预测集大小。我们展示了D-CP在定量和人体实验($n=120$)中的表现。我们的研究结果表明,CP预测集可以提高人类-AI团队的表现,而不是单独显示前1个预测,专家们发现D-CP预测集比CP预测集更有用。

英文摘要:

Research on human-AI teams usually provides experts with a single label, which ignores the uncertainty in a model's recommendation. Conformal prediction (CP) is a well established line of research that focuses on building a theoretically grounded, calibrated prediction set, which may contain multiple labels. We explore how such prediction sets impact expert decision-making in human-AI teams. Our evaluation on human subjects finds that set valued predictions positively impact experts. However, we notice that the predictive sets provided by CP can be very large, which leads to unhelpful AI assistants. To mitigate this, we introduce D-CP, a method to perform CP on some examples and defer to experts. We prove that D-CP can reduce the prediction set size of non-deferred examples. We show how D-CP performs in quantitative and in human subject experiments ($n=120$). Our results suggest that CP prediction sets improve human-AI team performance over showing the top-1 prediction alone, and that experts find D-CP prediction sets are more useful than CP prediction sets.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑