arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2106.03357stat.MLcs.LG
收藏

评估最新分类模型的贝叶斯最优性

Evaluating State-of-the-Art Classification Models Against Bayes Optimality

University of California, Berkeley(加州大学伯克利分校);Salesforce Research(赛富时研究院);University of Illinois Urbana-Champaign(伊利诺伊大学厄巴纳-香槟分校)

Ryan Theisen, Huan Wang, Lav R. Varshney, Caiming Xiong, Richard Socher

摘要:

评估一个给定的数据驱动的分类问题的固有困难是建立绝对的基准和评估在该领域的进展是重要的。为此,要考虑的自然量是\emph{Bayes error},其测量对于给定数据分布理论上可实现的最佳分类误差。虽然通常是一个棘手的量,但我们证明了我们可以计算使用规范化流学习的生成模型的精确贝叶斯误差。我们的技术依赖于一个基本的结果,即贝叶斯误差在可逆变换下是不变的。因此,我们可以通过计算高斯基分布来计算学习流模型的精确贝叶斯误差,这可以使用Holmes-Diaconis-Ross积分有效地完成。此外,我们表明,通过改变学习的流模型的温度,我们可以生成与标准基准数据集非常相似的合成数据集,但几乎具有任何期望的贝叶斯误差。我们使用我们的方法对最先进的分类模型进行了彻底的调查,并发现在某些情况下(但不是所有情况下),这些模型能够获得非常接近最佳的准确性。最后,我们使用我们的方法来评估标准基准数据集的内在“硬度”,以及这些数据集中的类。

英文摘要:

Evaluating the inherent difficulty of a given data-driven classification problem is important for establishing absolute benchmarks and evaluating progress in the field. To this end, a natural quantity to consider is the \emph{Bayes error}, which measures the optimal classification error theoretically achievable for a given data distribution. While generally an intractable quantity, we show that we can compute the exact Bayes error of generative models learned using normalizing flows. Our technique relies on a fundamental result, which states that the Bayes error is invariant under invertible transformation. Therefore, we can compute the exact Bayes error of the learned flow models by computing it for Gaussian base distributions, which can be done efficiently using Holmes-Diaconis-Ross integration. Moreover, we show that by varying the temperature of the learned flow models, we can generate synthetic datasets that closely resemble standard benchmark datasets, but with almost any desired Bayes error. We use our approach to conduct a thorough investigation of state-of-the-art classification models, and find that in some -- but not all -- cases, these models are capable of obtaining accuracy very near optimal. Finally, we use our method to evaluate the intrinsic "hardness" of standard benchmark datasets, and classes within those datasets.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑