Vote'n'Rank:基于社会选择理论的基准评测修订
Vote'n'Rank: Revision of Benchmarking with Social Choice Theory
AI总结:
本文提出社会选择理论框架 Vote'n'Rank,用投票排名替代多任务基准中的平均得分聚合,以更稳健地处理缺失分数并识别最优系统。
摘要:
机器学习(ML)不同应用领域中最先进系统的开发由基准测试推动,这些基准塑造了从多个视角评估泛化能力的范式。尽管该范式正转向跨多样任务的更细粒度评估,但如何聚合性能这一微妙问题已受到社区的特别关注。通常,基准测试遵循未言明的功利主义原则,即依据系统在任务特定指标上的平均得分对其进行排名。这种聚合过程被视为一种次优的评估协议,可能制造了进步的错觉。本文提出 Vote'n'Rank,这是一个在社会选择理论原则下对多任务基准中的系统进行排名的框架。我们证明,该方法可被高效用于在若干 ML 子领域获得关于基准测试的新洞见,并在研发案例研究中识别性能最佳的系统。Vote'n'Rank 的流程比平均得分更稳健,同时能够处理缺失的性能分数,并确定系统成为获胜者的条件。
英文摘要:
The development of state-of-the-art systems in different applied areas of machine learning (ML) is driven by benchmarks, which have shaped the paradigm of evaluating generalisation capabilities from multiple perspectives. Although the paradigm is shifting towards more fine-grained evaluation across diverse tasks, the delicate question of how to aggregate the performances has received particular interest in the community. In general, benchmarks follow the unspoken utilitarian principles, where the systems are ranked based on their mean average score over task-specific metrics. Such aggregation procedure has been viewed as a sub-optimal evaluation protocol, which may have created the illusion of progress. This paper proposes Vote'n'Rank, a framework for ranking systems in multi-task benchmarks under the principles of the social choice theory. We demonstrate that our approach can be efficiently utilised to draw new insights on benchmarking in several ML sub-fields and identify the best-performing systems in research and development case studies. The Vote'n'Rank's procedures are more robust than the mean average while being able to handle missing performance scores and determine conditions under which the system becomes the winner.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。