arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2210.05769cs.LGcs.CL
收藏

Vote'n'Rank:基于社会选择理论的基准评测修订

Vote'n'Rank: Revision of Benchmarking with Social Choice Theory

HSE University(高等经济大学);SaluteDevices(SaluteDevices公司);University of Oxford(牛津大学);Artificial Intelligence Research Institute(人工智能研究院);Institute of Linguistics, RAS(俄罗斯科学院语言学研究所);Center for Information and Language Processing, LMU Munich(慕尼黑大学信息与语言处理中心)

Mark Rofin, Vladislav Mikhailov, Mikhail Florinskiy, Andrey Kravchenko, Elena Tutubalina, Tatiana Shavrina, Daniel Karabekyan, Ekaterina Artemova

AI总结:

本文提出社会选择理论框架 Vote'n'Rank,用投票排名替代多任务基准中的平均得分聚合,以更稳健地处理缺失分数并识别最优系统。

摘要:

机器学习(ML)不同应用领域中最先进系统的开发由基准测试推动,这些基准塑造了从多个视角评估泛化能力的范式。尽管该范式正转向跨多样任务的更细粒度评估,但如何聚合性能这一微妙问题已受到社区的特别关注。通常,基准测试遵循未言明的功利主义原则,即依据系统在任务特定指标上的平均得分对其进行排名。这种聚合过程被视为一种次优的评估协议,可能制造了进步的错觉。本文提出 Vote'n'Rank,这是一个在社会选择理论原则下对多任务基准中的系统进行排名的框架。我们证明,该方法可被高效用于在若干 ML 子领域获得关于基准测试的新洞见,并在研发案例研究中识别性能最佳的系统。Vote'n'Rank 的流程比平均得分更稳健,同时能够处理缺失的性能分数,并确定系统成为获胜者的条件。

英文摘要:

The development of state-of-the-art systems in different applied areas of machine learning (ML) is driven by benchmarks, which have shaped the paradigm of evaluating generalisation capabilities from multiple perspectives. Although the paradigm is shifting towards more fine-grained evaluation across diverse tasks, the delicate question of how to aggregate the performances has received particular interest in the community. In general, benchmarks follow the unspoken utilitarian principles, where the systems are ranked based on their mean average score over task-specific metrics. Such aggregation procedure has been viewed as a sub-optimal evaluation protocol, which may have created the illusion of progress. This paper proposes Vote'n'Rank, a framework for ranking systems in multi-task benchmarks under the principles of the social choice theory. We demonstrate that our approach can be efficiently utilised to draw new insights on benchmarking in several ML sub-fields and identify the best-performing systems in research and development case studies. The Vote'n'Rank's procedures are more robust than the mean average while being able to handle missing performance scores and determine conditions under which the system becomes the winner.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑