ICML 2023 排名实验:考察机器学习/人工智能同行评审中的作者自我评估
The ICML 2023 Ranking Experiment: Examining Author Self-Assessment in ML/AI Peer Review
AI总结:
本文基于ICML 2023实验,利用作者对自身多篇投稿的质量排名提出保序机制校准评审分数,证明其优于原始分数,并给出低风险应用方案。
摘要:
我们在2023年国际机器学习大会(ICML)的评审过程中开展了一项实验,要求有多篇投稿的作者根据感知质量对自己的论文进行排名。我们总共收到1,342份排名,每份来自不同的作者,涵盖2,592篇投稿。在本文中,我们对作者提供的排名如何可用于改进机器学习会议的同行评审过程进行了实证分析。我们重点关注保序机制(Isotonic Mechanism),该机制利用作者提供的排名来校准原始评审分数。我们的分析表明,这些经过排名校准的分数在估计真实“期望评审分数”方面,在平方误差和绝对误差指标上均优于原始评审分数。此外,我们提出了保序机制和作者提供的排名在同行评审中的若干谨慎、低风险的应用,包括支持高级领域主席监督领域主席的建议、协助论文奖项的评选,以及指导紧急审稿人的招募。
英文摘要:
We conducted an experiment during the review process of the 2023 International Conference on Machine Learning (ICML), asking authors with multiple submissions to rank their papers based on perceived quality. In total, we received 1,342 rankings, each from a different author, covering 2,592 submissions. In this paper, we present an empirical analysis of how author-provided rankings could be leveraged to improve peer review processes at machine learning conferences. We focus on the Isotonic Mechanism, which calibrates raw review scores using the author-provided rankings. Our analysis shows that these ranking-calibrated scores outperform the raw review scores in estimating the ground truth ``expected review scores'' in terms of both squared and absolute error metrics. Furthermore, we propose several cautious, low-risk applications of the Isotonic Mechanism and author-provided rankings in peer review, including supporting senior area chairs in overseeing area chairs' recommendations, assisting in the selection of paper awards, and guiding the recruitment of emergency reviewers.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。