没有损失就没有一致:同行评审中的学习与社会选择
No Agreement Without Loss: Learning and Social Choice in Peer Review
AI总结:
本文质疑 Noothigattu 等人提出并应用于 AAAI 2022 的同行评审聚合框架,通过最小化损失函数聚合评审者映射,发现公理权衡与放弃不切实际假设会导致方法不连续等负面结果。
摘要:
在同行评审系统中,评审者通常被要求评估投稿的多个特征,如技术质量或新颖性。每个预定义特征都会被赋予一个分数,评审者需基于这些分数提供总体的定量推荐。可以假设每位评审者都有自己从特征集合到推荐的映射,且不同评审者心中的映射各不相同。这引入了一种被称为“通约偏差”的任意性因素。本文讨论了由 Noothigattu、Shah 和 Procaccia 提出并被 AAAI 2022 会议组织者应用的框架。他们提出通过最小化某些损失函数来聚合评审者的映射,并从社会选择理论的意义上研究了该方法的公理性质。我们质疑了其工作中的若干结果与假设,并报告了多项负面结果。一方面,我们研究了所提公理之间的权衡,以及该方法恰当捕捉多数评审者一致意见的能力。另一方面,我们表明放弃某项不切实际的假设会产生巨大影响,包括导致该方法不连续。
英文摘要:
In peer review systems, reviewers are often asked to evaluate various features of submissions, such as technical quality or novelty. A score is given to each of the predefined features and based on these the reviewer has to provide an overall quantitative recommendation. It may be assumed that each reviewer has her own mapping from the set of features to a recommendation, and that different reviewers have different mappings in mind. This introduces an element of arbitrariness known as commensuration bias. In this paper we discuss a framework, introduced by Noothigattu, Shah and Procaccia, and then applied by the organizers of the AAAI 2022 conference. Noothigattu, Shah and Procaccia proposed to aggregate reviewer's mapping by minimizing certain loss functions, and studied axiomatic properties of this approach, in the sense of social choice theory. We challenge several of the results and assumptions used in their work and report a number of negative results. On the one hand, we study a trade-off between some of the axioms proposed and the ability of the method to properly capture agreements of the majority of reviewers. On the other hand, we show that dropping a certain unrealistic assumption has dramatic effects, including causing the method to be discontinuous.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。