闭环人在回路延迟管道的设计
Designing Closed Human-in-the-loop Deferral Pipelines
摘要：
在混合人机延迟框架中，分类器可以将不确定的情况推迟给人类决策者（他们自己往往也会犯错）。在同时训练这种分类器和延迟模型方面，以前的工作通常假设在训练期间可以访问oracle，以获得训练样本的真实类标签，但实际上通常没有这样的oracle。相比之下，我们考虑一个“封闭”的决策管道，其中在延迟中使用的相同的易犯错的人类决策者也提供训练标签。如何使用不完美和有偏见的人类专家标签来训练公平和准确的延期框架？我们的关键见解是，通过利用弱先验信息，我们可以将专家与输入示例相匹配，以确保所产生的延迟框架的公平性和准确性，即使在使用不完美和有偏见的专家代替地面真值标签时也是如此。理论分析和两个任务的评价都表明了我们方法的有效性。
英文摘要：
In hybrid human-machine deferral frameworks, a classifier can defer uncertain cases to human decision-makers (who are often themselves fallible). Prior work on simultaneous training of such classifier and deferral models has typically assumed access to an oracle during training to obtain true class labels for training samples, but in practice there often is no such oracle. In contrast, we consider a "closed" decision-making pipeline in which the same fallible human decision-makers used in deferral also provide training labels. How can imperfect and biased human expert labels be used to train a fair and accurate deferral framework? Our key insight is that by exploiting weak prior information, we can match experts to input examples to ensure fairness and accuracy of the resulting deferral framework, even when imperfect and biased experts are used in place of ground truth labels. The efficacy of our approach is shown both by theoretical analysis and by evaluation on two tasks.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
