Peer review is widely regarded as essential for advancing scientific research. However, reviewers may be biased by authors' prestige or other characteristics. Double-blind peer review, in which the authors' identities are masked from the reviewers, has been proposed as a way to reduce reviewer bias. Although intuitive, evidence for the effectiveness of double-blind peer review in reducing bias is limited and mixed. Here, we examine the effects of double-blind peer review on prestige bias by analyzing the peer review files of 5027 papers submitted to the International Conference on Learning Representations (ICLR), a top computer science conference that changed its reviewing policy from single-blind peer review to double-blind peer review in 2018. We find that after switching to double-blind review, the scores given to the most prestigious authors significantly decreased. However, because many of these papers were above the threshold for acceptance, the change did not affect paper acceptance decisions significantly. Nevertheless, we show that double-blind peer review may have improved the quality of the selections by limiting other (non-author-prestige) biases. Specifically, papers rejected in the single-blind format are cited more than those rejected under the double-blind format, suggesting that double-blind review better identifies poorer quality papers. Interestingly, an apparently unrelated change - the change of rating scale from 10 to 4 points - likely reduced prestige bias significantly, to an extent that affected papers' acceptance. These results provide some support for the effectiveness of double-blind review in reducing prestige bias, while opening new research directions on the impact of peer review formats.
CABAL: Multi-Agent Simulacra for Tracing the Effects of Collusive Bidding in Peer Review
University of Macau（澳门大学）；Mila – Québec AI Institute（米拉-魁北克人工智能研究所）；McGill University（麦吉尔大学）；King Abdullah University of Science and Technology（阿卜杜拉国王科技大学）；University of Electronic Science and Technology of China（电子科技大学）
Commitment Checklist: Auditing Author Commitments in Peer Review
AIST, Japan（日本国家信息技术研究所）；Ubiquitous Knowledge Processing Lab (UKP Lab), Department of Computer Science, TU Darmstadt and National Research Center for Applied Cybersecurity ATHENE, Germany（图宾根大学计算机科学系通用知识处理实验室及应用网络安全国家研究中心ATHENE德国）
Insights from the ICLR Peer Review and Rebuttal Process
LMU Munich（慕尼黑大学）；Technical University of Munich（慕尼黑工业大学）；Munich Center for Machine Learning（慕尼黑机器学习中心）；University of Southern California（南加州大学）；Paper Copilot；Cardiff University（卡迪夫大学）
