arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 1709.02418cs.LG
收藏

关于AUC的知识如何约束可能的真值标记集?

How Does Knowledge of the AUC Constrain the Set of Possible Ground-truth Labelings?

Worcester Polytechnic Institute(伍斯特理工学院)

Jacob Whitehill

摘要:

最近关于隐私保护的机器学习的工作已经考虑了Kaggle等数据挖掘竞赛如何可能被“黑客”,无论是有意还是无意,通过使用来自Oracle的信息,报告分类器在测试集上的准确性。特别是对于二进制分类任务,最常见的准确性指标之一是ROC曲线下面积(AUC),在本文中,我们探讨了如何从实值“猜测”的n向量相对于真值标签计算AUC的数学结构。我们展示了如何在测试集上的分类器的AUC的知识可以约束一组可能的真值标签,我们推导出一个算法来计算这些标签的确切数量,并有效地枚举它们。最后,我们提供的经验证据表明,令人惊讶的是,兼容的标签的数量实际上可以随着n的增长而减少,直到达到测试集相关的阈值。

英文摘要:

Recent work on privacy-preserving machine learning has considered how data-mining competitions such as Kaggle could potentially be "hacked", either intentionally or inadvertently, by using information from an oracle that reports a classifier's accuracy on the test set. For binary classification tasks in particular, one of the most common accuracy metrics is the Area Under the ROC Curve (AUC), and in this paper we explore the mathematical structure of how the AUC is computed from an n-vector of real-valued "guesses" with respect to the ground-truth labels. We show how knowledge of a classifier's AUC on the test set can constrain the set of possible ground-truth labelings, and we derive an algorithm both to compute the exact number of such labelings and to enumerate efficiently over them. Finally, we provide empirical evidence that, surprisingly, the number of compatible labelings can actually decrease as n grows, until a test set-dependent threshold is reached.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑