利用在机器学习竞赛中报告AUC分数的Oracle
Exploiting an Oracle that Reports AUC Scores in Machine Learning Contests
摘要:
在机器学习竞赛中,如ImageNet大规模视觉识别挑战赛和KDD Cup,参赛者可以提交候选解决方案,并从Oracle(通常是竞赛的组织者)那里获得与地面事实标签相比的猜测准确性。二元分类任务最常用的准确性度量之一是受试者工作特征曲线下面积(AUC)。在本文中,我们提供了概念证明的知识的AUC的一组猜测可以使用,在两种不同的攻击,以提高这些猜测的准确性。另一方面,我们还证明了一种AUC利用的棘手性,证明了候选解决方案获得$c$ AUC得分的$n$示例的可能二进制标记的数量在$n$中呈指数增长,对于每个$c\in (0,1)$。
英文摘要:
In machine learning contests such as the ImageNet Large Scale Visual Recognition Challenge and the KDD Cup, contestants can submit candidate solutions and receive from an oracle (typically the organizers of the competition) the accuracy of their guesses compared to the ground-truth labels. One of the most commonly used accuracy metrics for binary classification tasks is the Area Under the Receiver Operating Characteristics Curve (AUC). In this paper we provide proofs-of-concept of how knowledge of the AUC of a set of guesses can be used, in two different kinds of attacks, to improve the accuracy of those guesses. On the other hand, we also demonstrate the intractability of one kind of AUC exploit by proving that the number of possible binary labelings of $n$ examples for which a candidate solution obtains a AUC score of $c$ grows exponentially in $n$, for every $c\in (0,1)$.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。