We consider three important challenges in conference peer review: (i) reviewers maliciously attempting to get assigned to certain papers to provide positive reviews, possibly as part of quid-pro-quo arrangements with the authors; (ii) "torpedo reviewing," where reviewers deliberately attempt to get assigned to certain papers that they dislike in order to reject them; (iii) reviewer de-anonymization on release of the similarities and the reviewer-assignment code. On the conceptual front, we identify connections between these three problems and present a framework that brings all these challenges under a common umbrella. We then present a (randomized) algorithm for reviewer assignment that can optimally solve the reviewer-assignment problem under any given constraints on the probability of assignment for any reviewer-paper pair. We further consider the problem of restricting the joint probability that certain suspect pairs of reviewers are assigned to certain papers, and show that this problem is NP-hard for arbitrary constraints on these joint probabilities but efficiently solvable for a practical special case. Finally, we experimentally evaluate our algorithms on datasets from past conferences, where we observe that they can limit the chance that any malicious reviewer gets assigned to their desired paper to 50% while producing assignments with over 90% of the total optimal similarity. Our algorithms still achieve this similarity while also preventing reviewers with close associations from being assigned to the same paper.
Paper Quality Assessment based on Individual Wisdom Metrics from Open Peer Review
Department of Electrical Engineering and Computer Science, MIT(麻省理工学院电气工程与计算机科学系);Département de Psychologie, Université de Montréal, Canada(蒙特利尔大学心理学系);Department of Experimental Psychology, University of Oxford, United Kingdom(牛津大学实验心理学系);Department of Brain and Cognitive Sciences & McGovern Institute, MIT(麻省理工学院脑科学与认知科学系)
ReviewerToo: Should AI Join The Program Committee? A Look At The Future of Peer Review
Mila – Quebec AI Institute(魁北克AI研究所);HEC Montréal(蒙特利尔高等商学院);ServiceNow Research(ServiceNow研究);Université de Montréal(蒙特利尔大学);Canada CIFAR Chair(加拿大 CIFAR 席位);Polytechnique Montréal(蒙特利尔理工学院)
本文提出 ReviewerToo 框架,用于研究 AI 辅助同行评审,在 ICLR 2025 数据集上验证其性能接近人类平均水平,并指出 AI 的优势与局限,提出集成指南,以提升评审的一致性和公平性。
No Agreement Without Loss: Learning and Social Choice in Peer Review
Institute for Mathematical and Computational Engineering, PUC(智利天主教大学数学与计算工程研究所);Instituto Milenio Fundamentos de los Datos(千禧数据基础研究所);Centro Nacional de Inteligencia Artificial(国家人工智能中心);Universidad Andres Bello(安德烈斯·贝略大学);Institute of Fundamental Technological Research, Polish Academy of Sciences(波兰科学院基础技术研究所)
Insights from the ICLR Peer Review and Rebuttal Process
LMU Munich(慕尼黑大学);Technical University of Munich(慕尼黑工业大学);Munich Center for Machine Learning(慕尼黑机器学习中心);University of Southern California(南加州大学);Paper Copilot;Cardiff University(卡迪夫大学)
CABAL: Multi-Agent Simulacra for Tracing the Effects of Collusive Bidding in Peer Review
University of Macau(澳门大学);Mila – Québec AI Institute(米拉-魁北克人工智能研究所);McGill University(麦吉尔大学);King Abdullah University of Science and Technology(阿卜杜拉国王科技大学);University of Electronic Science and Technology of China(电子科技大学)
When Reviews Disagree: Fine-Grained Contradiction Analysis in Scientific Peer Reviews
Department of Computer Science and Engineering, Indian Institute of Technology Patna, India(印度理工学院帕纳瓦分校计算机科学与工程系);School of Computer Engineering, KIIT Deemed to be University, Bhubaneswar, India(比哈尔邦布尔萨大学计算机工程学院)
FactReview: Evidence-Grounded Peer Review with Execution-Based Claim Verification
Rensselaer Polytechnic Institute(罗切斯特理工学院);Sun Yat-sen University(中山大学);Southeast University(东南大学);The Hong Kong University of Science and Technology(香港科技大学)
Pre-review to Peer review: Pitfalls of Automating Reviews using Large Language Models
AllSci Corp(AllSci公司);Sunwater Capital(Sunwater资本);Kellogg School of Management, Northwestern University(西北大学凯洛格管理学院);Dept. of Computer Science, Northern Illinois University(北伊利诺伊大学计算机科学系);Institute of Foundation Models, MBZUAI(基础模型研究所,MBZUAI)
FMMD: A multimodal multidisciplinary dataset of open peer reviews from F1000Research
School of Computer Science and Engineering, Guangzhou Institute of Science and Technology(计算机科学与工程学院,广州科学与技术研究所);College of Foreign Languages and Cultures, Xiamen University(外语学院,厦门大学);Science and Education Evaluation Lab, Guangzhou Institute of Science and Technology(科学与教育评估实验室,广州科学与技术研究所)
Can LLM feedback enhance review quality? A randomized study of 20K reviews at ICLR 2025
Stanford University(斯坦福大学);Georgia Institute of Technology(佐治亚理工学院);University of California, Los Angeles(加利福尼亚大学洛杉矶分校);Google Research(谷歌研究院);University of California, San Diego(加利福尼亚大学圣迭戈分校);Columbia University(哥伦比亚大学)