arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2108.13034cs.LG
收藏

基于FeeBee的真实数据集Bayes误差估计

Evaluating Bayes Error Estimators on Real-World Datasets with FeeBee

ETH Zurich(苏黎世联邦理工学院);University of Copenhagen(哥本哈根大学)

Cedric Renggli, Luka Rimanic, Nora Hollenstein, Ce Zhang

摘要:

贝叶斯错误率(BER)是机器学习中的一个基本概念,它量化了任何分类器在固定概率分布上所能达到的最佳准确度。尽管多年来一直在研究BER的上下界估计,但这些估计通常只在具有已知概率分布的合成数据集上进行比较,留下两个关键问题没有回答:(1)它们在真实世界数据集上的表现如何?(2)实用性如何?这些都不是小事。除了现实世界数据集的未知BER的明显挑战外,任何BER估计器都需要克服两个主要方面,以便适用于现实世界的设置:(1)计算和样本复杂性,以及(2)超参数的敏感性和选择。在这项工作中,我们提出了FeeBee,这是第一个原则性框架,用于分析和比较具有未知概率分布的任何现代现实数据集的BER估计器。我们实现这一点,通过注入一个控制量的标签噪声和执行一系列不同的噪声水平上的多个评估,支持的理论结果,允许得出结论的BER的演变。通过在计算机视觉和NLP领域的6个常用数据集上实现和分析7个多类BER估计器,FeeBee可以对这些估计器进行深入研究,清楚地识别每个估计器的优点和缺点,同时可以轻松地部署在任何未来的BER估计器上。

英文摘要:

The Bayes error rate (BER) is a fundamental concept in machine learning that quantifies the best possible accuracy any classifier can achieve on a fixed probability distribution. Despite years of research on building estimators of lower and upper bounds for the BER, these were usually compared only on synthetic datasets with known probability distributions, leaving two key questions unanswered: (1) How well do they perform on real-world datasets?, and (2) How practical are they? Answering these is not trivial. Apart from the obvious challenge of an unknown BER for real-world datasets, there are two main aspects any BER estimator needs to overcome in order to be applicable in real-world settings: (1) the computational and sample complexity, and (2) the sensitivity and selection of hyper-parameters. In this work, we propose FeeBee, the first principled framework for analyzing and comparing BER estimators on any modern real-world dataset with unknown probability distribution. We achieve this by injecting a controlled amount of label noise and performing multiple evaluations on a series of different noise levels, supported by a theoretical result which allows drawing conclusions about the evolution of the BER. By implementing and analyzing 7 multi-class BER estimators on 6 commonly used datasets of the computer vision and NLP domains, FeeBee allows a thorough study of these estimators, clearly identifying strengths and weaknesses of each, whilst being easily deployable on any future BER estimator.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑