We consider peer review in a conference setting where there is typically an overlap between the set of reviewers and the set of authors. This overlap can incentivize strategic reviews to influence the final ranking of one's own papers. In this work, we address this problem through the lens of social choice, and present a theoretical framework for strategyproof and efficient peer review. We first present and analyze an algorithm for reviewer-assignment and aggregation that guarantees strategyproofness and a natural efficiency property called unanimity, when the authorship graph satisfies a simple property. Our algorithm is based on the so-called partitioning method, and can be thought as a generalization of this method to conference peer review settings. We then empirically show that the requisite property on the authorship graph is indeed satisfied in the submission data from the ICLR conference, and further demonstrate a simple trick to make the partitioning method more practically appealing for conference peer review. Finally, we complement our positive results with negative theoretical results where we prove that under various ways of strengthening the requirements, it is impossible for any algorithm to be strategyproof and efficient.
Strategyproof Peer Selection using Randomization, Partitioning, and Apportionment
UNSW Sydney(新南威尔士大学);Data61 CSIRO(联邦科学与工业研究组织Data61);Ben-Gurion University of the Negev(内盖夫本-古里安大学);Tulane University(杜兰大学);Hebrew University of Jerusalem(耶路撒冷希伯来大学)
How to Find Fantastic AI Papers: Self-Rankings as a Powerful Predictor of Scientific Impact Beyond Peer Review
University of Pennsylvania(宾夕法尼亚大学);Yale University(耶鲁大学);University of North Carolina at Chapel Hill(北卡罗来纳大学教堂山分校);New York University(纽约大学);Princeton University(普林斯顿大学)
CABAL: Multi-Agent Simulacra for Tracing the Effects of Collusive Bidding in Peer Review
University of Macau(澳门大学);Mila – Québec AI Institute(米拉-魁北克人工智能研究所);McGill University(麦吉尔大学);King Abdullah University of Science and Technology(阿卜杜拉国王科技大学);University of Electronic Science and Technology of China(电子科技大学)
No Agreement Without Loss: Learning and Social Choice in Peer Review
Institute for Mathematical and Computational Engineering, PUC(智利天主教大学数学与计算工程研究所);Instituto Milenio Fundamentos de los Datos(千禧数据基础研究所);Centro Nacional de Inteligencia Artificial(国家人工智能中心);Universidad Andres Bello(安德烈斯·贝略大学);Institute of Fundamental Technological Research, Polish Academy of Sciences(波兰科学院基础技术研究所)
When Reviews Disagree: Fine-Grained Contradiction Analysis in Scientific Peer Reviews
Department of Computer Science and Engineering, Indian Institute of Technology Patna, India(印度理工学院帕纳瓦分校计算机科学与工程系);School of Computer Engineering, KIIT Deemed to be University, Bhubaneswar, India(比哈尔邦布尔萨大学计算机工程学院)
The ICML 2023 Ranking Experiment: Examining Author Self-Assessment in ML/AI Peer Review
University of Pennsylvania(宾夕法尼亚大学);Massachusetts Institute of Technology(麻省理工学院);University of North Carolina at Chapel Hill(北卡罗来纳大学教堂山分校);New York University(纽约大学);Princeton University(普林斯顿大学)
Paper Quality Assessment based on Individual Wisdom Metrics from Open Peer Review
Department of Electrical Engineering and Computer Science, MIT(麻省理工学院电气工程与计算机科学系);Département de Psychologie, Université de Montréal, Canada(蒙特利尔大学心理学系);Department of Experimental Psychology, University of Oxford, United Kingdom(牛津大学实验心理学系);Department of Brain and Cognitive Sciences & McGovern Institute, MIT(麻省理工学院脑科学与认知科学系)