Peer review often involves reviewers submitting their independent reviews, followed by a discussion among reviewers of each paper. A question among policymakers is whether the reviewers of a paper should be anonymous to each other during the discussion. We shed light on this by conducting a randomized controlled trial at the UAI 2022 conference. We randomly split the reviewers and papers into two conditions--one with anonymous discussions and the other with non-anonymous discussions, and conduct an anonymous survey of all reviewers, to address the following questions: 1. Do reviewers discuss more in one of the conditions? Marginally more in anonymous (n = 2281, p = 0.051). 2. Does seniority have more influence on final decisions when non-anonymous? Yes, the decisions are closer to senior reviewers' scores in the non-anonymous condition than in anonymous (n = 484, p = 0.04). 3. Are reviewers more polite in one of the conditions? No significant difference in politeness of reviewers' text-based responses (n = 1125, p = 0.72). 4. Do reviewers' self-reported experiences differ across the two conditions? No significant difference for each of the five questions asked (n = 132 and p > 0.3). 5. Do reviewers prefer one condition over the other? Yes, there is a weak preference for anonymous discussions (n = 159 and Cohen's d= 0.25). 6. What do reviewers consider important to make policy on anonymity among reviewers? Reviewers' feeling of safety in expressing their opinions was rated most important, while polite communication among reviewers was rated least important (n = 159). 7. Have reviewers experienced dishonest behavior due to non-anonymity in discussions? Yes, roughly 7% of respondents answered affirmatively (n = 167). Overall, this experiment reveals evidence supporting an anonymous discussion setup in the peer-review process, in terms of the evaluation criteria considered.
Insights from the ICLR Peer Review and Rebuttal Process
LMU Munich（慕尼黑大学）；Technical University of Munich（慕尼黑工业大学）；Munich Center for Machine Learning（慕尼黑机器学习中心）；University of Southern California（南加州大学）；Paper Copilot；Cardiff University（卡迪夫大学）
Center for Advanced Studies in Bioscience Innovation Law (CeBIL), Faculty of Law, University of Copenhagen（哥本哈根大学法学院生物科学创新法律高级研究中心）；Center for Biomedical Ethics, Yong Loo Lin School of Medicine, National University of Singapore（新加坡国立大学杨潞龄医学院生物医学伦理中心）；Faculty of Law, University of Cambridge（剑桥大学法学院）；Department of Psychology, Yonsei University, Seoul（延世大学心理学系）；Vincent V.C. Woo Chinese Medicine Clinical Research Institute, School of Chinese Medicine, Hong Kong Baptist University（香港浸会大学中医学院吴振圭中医临床研究所）；Chinese EQUATOR Centre, Hong Kong（中国EQUATOR中心）；School of Allied and Community Health, London South Bank University, London（伦敦南岸大学 allied 与社区健康学院）；Uehiro Oxford Institute, University of Oxford, Oxford（牛津大学威希罗牛津研究所）
CABAL: Multi-Agent Simulacra for Tracing the Effects of Collusive Bidding in Peer Review
University of Macau（澳门大学）；Mila – Québec AI Institute（米拉-魁北克人工智能研究所）；McGill University（麦吉尔大学）；King Abdullah University of Science and Technology（阿卜杜拉国王科技大学）；University of Electronic Science and Technology of China（电子科技大学）
Pre-review to Peer review: Pitfalls of Automating Reviews using Large Language Models
AllSci Corp（AllSci公司）；Sunwater Capital（Sunwater资本）；Kellogg School of Management, Northwestern University（西北大学凯洛格管理学院）；Dept. of Computer Science, Northern Illinois University（北伊利诺伊大学计算机科学系）；Institute of Foundation Models, MBZUAI（基础模型研究所，MBZUAI）
LLM-as-a-Reviewer: Benchmarking Their Ability, Divergence, and Prompt Injection Resistance as Paper Reviewers
University of South Florida（佛罗里达南大学）；Missouri University of Science and Technology（密苏里科技大学）；University of Alabama（阿拉巴马大学）；Florida International University（佛罗里达国际大学）；University of Cincinnati（辛辛那提大学）；George Mason University（乔治·梅森大学）
Reviewing the Reviewer: LLM-Assisted Reviewer Feedback Generation for Guideline Compliance
Ubiquitous Knowledge Processing Lab (UKP Lab), Department of Computer Science, Technical University of Darmstadt and National Research Center for Applied Cybersecurity ATHENE, Germany（技术大学达姆施塔特计算机科学系 Ubiquitous Knowledge Processing Lab（UKP Lab）及国家应用网络安全研究中心 ATHENE，德国）；Department of Data Science & AI, Monash University, Australia（墨尔本大学数据科学与人工智能系，澳大利亚）；Data Science Group, University of Hamburg（汉堡大学数据科学组）
