Automated AI classifiers should be able to defer the prediction to a human decision maker to ensure more accurate predictions. In this work, we jointly train a classifier with a rejector, which decides on each data point whether the classifier or the human should predict. We show that prior approaches can fail to find a human-AI system with low misclassification error even when there exists a linear classifier and rejector that have zero error (the realizable setting). We prove that obtaining a linear pair with low error is NP-hard even when the problem is realizable. To complement this negative result, we give a mixed-integer-linear-programming (MILP) formulation that can optimally solve the problem in the linear setting. However, the MILP only scales to moderately-sized problems. Therefore, we provide a novel surrogate loss function that is realizable-consistent and performs well empirically. We test our approaches on a comprehensive set of datasets and compare to a wide range of baselines.
Incorporating Uncertainty in Learning to Defer Algorithms for Safe Computer-Aided Diagnosis
Centre for Big Data Research in Health, University of New South Wales(新南威尔士大学健康大数据研究中心)
深度神经网络越来越多地用于计算机辅助诊断,但错误的诊断可能会给患者带来极其高昂的代价。我们提出了一个学习推迟与不确定性(LDU)算法,识别患者的诊断不确定性高,并推迟他们的评估由人类专家。评价LDU对心肌梗死的诊断(使用出院总结)、任何合并症的诊断(使用结构化数据)以及胸腔积液和气胸的诊断(使用胸部X线片),并与“学习在没有不确定信息的情况下推迟”(LD)和“不确定性直接分诊”(DT)方法进行比较。LDU达到了与LD相同的F1评分,但延迟的患者数量显著减少(例如,对于诊断胸腔积液,F1评分为0.96,延迟率为36% vs. 69%)。此外,即使许多患者被分配了具有高置信度的错误诊断(例如,对于任何合并症的诊断),LDU的F1评分也增加了17%,而DT不适用。重要的是,LDU中延迟损失的权重可以很容易地调整,以获得诊断准确性和延迟率之间的理想权衡。总之,LDU可以很容易地增强任何现有的诊断网络,以降低临床实践中误诊的风险。
Learning to Defer for Causal Discovery with Imperfect Experts
University of Oxford(牛津大学);Mila – Quebec Artificial Intelligence Institute(米拉-魁北克人工智能研究所);Université de Montréal(蒙特利尔大学);ServiceNow Research(ServiceNow研究院);University of Amsterdam(阿姆斯特丹大学);Université Laval(拉瓦尔大学)
Beyond Augmented-Action Surrogates for Multi-Expert Learning-to-Defer
School of Computing(计算学院);Fédération ENAC(ENAC联合会);ISAE-SUPAERO;ONERA;National University of Singapore(新加坡国立大学);Université de Toulouse(图卢兹大学);Agency for Science, Technology and Research(科技研究局);Institute for Infocomm Research(信息通信研究所)
When No-Rejection Learning is Consistent for Regression with Rejection
Imperial College Business School, Imperial College London(伦敦帝国学院商学院);Institute for Computational and Mathematical Engineering, Stanford University(斯坦福大学计算与数学工程研究所)