arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2202.03673cs.LGstat.ML
收藏

使用One-vs-All分类器的校准学习延迟

Calibrated Learning to Defer with One-vs-All Classifiers

University of Amsterdam(阿姆斯特丹大学)

Rajeev Verma, Eric Nalisnick

摘要:

学习延迟(L2 D)框架有可能使AI系统更安全。对于给定的输入,如果人类比模型更有可能采取正确的行动,则系统可以将决策推迟给人类。我们研究了L2 D系统的校准,调查它们输出的概率是否合理。我们发现,Mozannar & Sontag(2020)的多类框架没有根据专家的正确性进行校准。此外,它甚至不能保证产生有效的概率,因为它的参数化为此目的而退化。我们提出了一个L2 D系统的基础上,一对所有的分类器,能够产生校准的专家正确性的概率。此外,我们的损失函数也是多类L2 D的一致替代,如Mozannar & Sontag(2020)。我们的实验证明,我们的系统不仅经过校准,而且这种好处不会以精度为代价。我们的模型的准确性始终与Mozannar & Sontag(2020)模型的准确性相当(并且通常更好),从仇恨言论检测到星系分类再到皮肤病变诊断。

英文摘要:

The learning to defer (L2D) framework has the potential to make AI systems safer. For a given input, the system can defer the decision to a human if the human is more likely than the model to take the correct action. We study the calibration of L2D systems, investigating if the probabilities they output are sound. We find that Mozannar & Sontag's (2020) multiclass framework is not calibrated with respect to expert correctness. Moreover, it is not even guaranteed to produce valid probabilities due to its parameterization being degenerate for this purpose. We propose an L2D system based on one-vs-all classifiers that is able to produce calibrated probabilities of expert correctness. Furthermore, our loss function is also a consistent surrogate for multiclass L2D, like Mozannar & Sontag's (2020). Our experiments verify that not only is our system calibrated, but this benefit comes at no cost to accuracy. Our model's accuracy is always comparable (and often superior) to Mozannar & Sontag's (2020) model's in tasks ranging from hate speech detection to galaxy classification to diagnosis of skin lesions.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑