arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2411.00328stat.MLcs.LG
收藏

我们需要多少个分类器?

How many classifiers do we need?

University of California, Berkeley(加利福尼亚大学伯克利分校);University of Melbourne(墨尔本大学);Harmonic Discovery(谐波发现公司);ICSI(国际计算机科学研究所);LBNL(劳伦斯伯克利国家实验室)

Hyunsuk Kim, Liam Hodgkinson, Ryan Theisen, Michael W. Mahoney

AI总结:

针对数据与模型规模扩容收益递减下的集成分类性能提升问题,本文提出极化度概念与神经极化定律,推导多数投票误差紧上界与分歧度渐近行为,可用于预测多分类器集成性能,且经图像分类实验验证。

摘要:

当通过扩大数据规模和/或模型尺寸带来的性能提升出现收益递减时,转向集成学习的做法正变得越来越流行——集成学习将多个模型的预测结果结合起来以提高准确率。本文针对分类任务中的多数投票策略,详细分析了分类器之间的分歧度与极化度(本文提出并定义的概念)和聚合单个分类器所获得的性能提升之间的关联。我们通过以下方式解答这些问题:(1)推导了极化度的上界,并提出我们所称的神经极化定律:大多数插值神经网络模型具有4/3极化特性。我们的实验结果不仅支持这一猜想,还表明对于一个数据集而言,极化度几乎是恒定的,与分类器的超参数或架构无关。(2)在受限熵条件下研究了多数投票分类器的误差,提出了一个紧上界,该上界表明分歧度与目标值线性相关,且斜率与极化度呈线性关系。(3)证明了分歧度随分类器数量变化的渐近行为结果,表明该结果有助于从较少分类器的性能预测更多分类器的性能。我们的理论和结论得到了多种神经网络在多个图像分类任务上的实验结果的支持。

英文摘要:

As performance gains through scaling data and/or model size experience diminishing returns, it is becoming increasingly popular to turn to ensembling, where the predictions of multiple models are combined to improve accuracy. In this paper, we provide a detailed analysis of how the disagreement and the polarization (a notion we introduce and define in this paper) among classifiers relate to the performance gain achieved by aggregating individual classifiers, for majority vote strategies in classification tasks. We address these questions in the following ways. (1) An upper bound for polarization is derived, and we propose what we call a neural polarization law: most interpolating neural network models are 4/3-polarized. Our empirical results not only support this conjecture but also show that polarization is nearly constant for a dataset, regardless of hyperparameters or architectures of classifiers. (2) The error of the majority vote classifier is considered under restricted entropy conditions, and we present a tight upper bound that indicates that the disagreement is linearly correlated with the target, and that the slope is linear in the polarization. (3) We prove results for the asymptotic behavior of the disagreement in terms of the number of classifiers, which we show can help in predicting the performance for a larger number of classifiers from that of a smaller number. Our theories and claims are supported by empirical results on several image classification tasks with various types of neural networks.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑