不要两次标注：在预算下比较二分类器时数量胜过质量
Don't Label Twice: Quantity Beats Quality when Comparing Binary Classifiers on a Budget
AI总结：
研究如何在预算内有效利用噪声标签比较两个二分类器的准确性，证明在选择更优分类器时，应优先收集更多样本的单一标签而非多个标签的多数投票。
摘要：
我们研究如何在预算内有效利用噪声标签来比较两个二分类器的准确性。通常的做法是通过多数投票将给定数据点的多个噪声标签汇总成一个更干净的标签。我们证明了一个与传统智慧相悖的定理。如果目标是确定两个分类器中更好的一个，我们证明最好将预算用于收集更多样本的单一标签。我们的结果源于Cramér定理在大偏差理论中的非平凡应用。我们讨论了我们的工作对机器学习基准设计的影响，其中推翻了一些历史悠久的建议。此外，我们的结果提供了优于Hoeffding界限的样本大小界限。
英文摘要：
We study how to best spend a budget of noisy labels to compare the accuracy of two binary classifiers. It's common practice to collect and aggregate multiple noisy labels for a given data point into a less noisy label via a majority vote. We prove a theorem that runs counter to conventional wisdom. If the goal is to identify the better of two classifiers, we show it's best to spend the budget on collecting a single label for more samples. Our result follows from a non-trivial application of Cramér's theorem, a staple in the theory of large deviations. We discuss the implications of our work for the design of machine learning benchmarks, where they overturn some time-honored recommendations. In addition, our results provide sample size bounds superior to what follows from Hoeffding's bound.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
