arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2303.07272stat.MEcs.LG
收藏

发明抛硬币分类器:考量机器学习基准性能中的多重性问题

Inventing a coin-flip classifier: Accounting for multiplicity in machine learning benchmark performance

Sami University of Applied Sciences, Department of Computer Science UiT — The Arctic University of Norway(萨米应用科学大学计算机科学系UiT — 北极大学)

Kajsa Møllersen, Einar Holsbø

AI总结:

该研究指出当前机器学习SOTA性能是样本最大值估计,提出应通过最佳分类器的预期性能估计SOTA,正式区分两种估计方式并给出相关分布,结合公开挑战赛案例分析了误用的实际影响。

摘要:

最先进(SOTA)性能指某一模型在测试样本上取得的最高性能,理想情况下是在公开数据(可复现性)或公开挑战赛(独立样本)等受控条件下得到的。数千个分类器被投入应用,其中的最高性能会成为特定问题的新参考基准。实际上,这种设置是对应用于随机样本的所有分类器的预期最佳性能的估计,即一种样本最大值估计。本文提出,SOTA 反而应当通过最佳分类器的预期性能来估计,且无需知晓具体哪个分类器是最佳的。我们的贡献在于正式区分了这两种估计方式,并探究了用前者估计后者的实际影响。我们通过给出非独立同分布且存在依赖关系的分类器的样本最大值估计量分布展开研究,还借助公开挑战赛中的真实案例阐释了其影响。

英文摘要:

State-of-the-art (SOTA) performance refers to the highest performance achieved by some model on a test sample, preferably under controlled conditions such as public data (reproducibility) or public challenges (independent sample). Thousands of classifiers are applied, and the highest performance becomes the new reference point for a particular problem. In effect, this set-up is an estimate of the expected best performance among all classifiers applied to a random sample; a sample maximum estimate. In this paper, we argue that SOTA should instead be estimated by the expected performance of the best classifier, which can be done without knowing which classifier it is. Our contribution is the formal distinction between the two, and an investigation into the practical consequences of using the former to estimate the latter. This is done by presenting sample maximum estimator distributions for non-identical and dependent classifiers. We illustrate the impact on real world examples from public challenges.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑