谁是胜出的算法?用于比较研究的排名聚合
Who is the Winning Algorithm? Rank Aggregation for Comparative Studies
AI总结:
本文提出了一种新的框架,通过利用完整排名信息来提高算法胜出概率的估计,从而更准确地识别在未见数据集上表现最佳的算法。
摘要:
考虑一组m个竞争的机器学习算法。在给定它们在基准数据集上的性能后,我们希望识别表现最佳的算法。具体而言,哪一个算法最有可能在未来的未见数据集上“胜出”(排名最高)。标准的最大似然方法建议统计每个算法的胜利次数。在本文中,我们认为完整排名中包含更多信息。也就是说,每个算法第二次、第三次等结束的次数。然而,如何有效地利用这些信息尚不明确。在本文中,我们引入了一个新的概念框架,用于根据基准数据集上的完整排名估计每个m个算法的胜出概率。我们提出的框架在合成和现实世界示例中显著优于目前已知的方法。
英文摘要:
Consider a collection of m competing machine learning algorithms. Given their performance on a benchmark of datasets, we would like to identify the best performing algorithm. Specifically, which algorithm is most likely to ``win'' (rank highest) on a future, unseen dataset. The standard maximum likelihood approach suggests counting the number of wins per each algorithm. In this work, we argue that there is much more information in the complete rankings. That is, the number of times that each algorithm finished second, third and so forth. Yet, it is not entirely clear how to effectively utilize this information for our purpose. In this work we introduce a novel conceptual framework for estimating the win probability for each of the m algorithms, given their complete rankings over a benchmark of datasets. Our proposed framework significantly improves upon currently known methods in synthetic and real-world examples.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。