Peer review is a core element of the scientific process, particularly in conference-centered fields such as ML and NLP. However, only few studies have evaluated its properties empirically. Aiming to fill this gap, we present a corpus that contains over 4k reviews and 1.2k author responses from ACL-2018. We quantitatively and qualitatively assess the corpus. This includes a pilot study on paper weaknesses given by reviewers and on quality of author responses. We then focus on the role of the rebuttal phase, and propose a novel task to predict after-rebuttal (i.e., final) scores from initial reviews and author responses. Although author responses do have a marginal (and statistically significant) influence on the final scores, especially for borderline papers, our results suggest that a reviewer's final score is largely determined by her initial score and the distance to the other reviewers' initial scores. In this context, we discuss the conformity bias inherent to peer reviewing, a bias that has largely been overlooked in previous research. We hope our analyses will help better assess the usefulness of the rebuttal phase in NLP conferences.
Insights from the ICLR Peer Review and Rebuttal Process
LMU Munich（慕尼黑大学）；Technical University of Munich（慕尼黑工业大学）；Munich Center for Machine Learning（慕尼黑机器学习中心）；University of Southern California（南加州大学）；Paper Copilot；Cardiff University（卡迪夫大学）
Monitoring AI-Modified Content at Scale: A Case Study on the Impact of ChatGPT on AI Conference Peer Reviews
Department of Computer Science, Stanford University（斯坦福大学计算机科学系）；Machine Learning Department, NEC Labs America（NEC美国实验室机器学习部门）；Department of Biomedical Data Science, Stanford University（斯坦福大学生物医学数据科学系）；Department of Electrical Engineering, Stanford University（斯坦福大学电气工程系）；Graduate School of Education, Stanford University（斯坦福大学教育研究生院）；Department of Sociology, Stanford University（斯坦福大学社会学系）；Graduate School of Business, Stanford University（斯坦福大学商学院）；Department of Management Science and Engineering, Stanford University（斯坦福大学管理科学与工程系）；Department of Computer Science, UC Santa Barbara（加州大学圣芭芭拉分校计算机科学系）
When Large Language Models Meet Law: Dual-Lens Taxonomy, Technical Advances, and Ethical Governance
School of Law, China University of Political Science and Law（中国政法大学法学院）；Zhejiang University of Finance and Economics Dongfang College（浙江财经大学东洋学院）；School of Information Management for Law, China University of Political Science and Law（中国政法大学信息管理法学院）；Department of Artificial Intelligence, Chung-Ang University（成均馆大学人工智能系）；Department of Data Science and Artificial Intelligence, The Hong Kong Polytechnic University（香港理工大学数据科学与人工智能系）
Can LLM feedback enhance review quality? A randomized study of 20K reviews at ICLR 2025
Stanford University（斯坦福大学）；Georgia Institute of Technology（佐治亚理工学院）；University of California, Los Angeles（加利福尼亚大学洛杉矶分校）；Google Research（谷歌研究院）；University of California, San Diego（加利福尼亚大学圣迭戈分校）；Columbia University（哥伦比亚大学）
Reproducibility in NLP: What Have We Learned from the Checklist?
Allen Institute for Artificial Intelligence（艾伦人工智能研究所）；Paul G. Allen School of Computer Science & Engineering, University of Washington（华盛顿大学保罗·G·艾伦计算机科学与工程学院）
NovBench: Evaluating Large Language Models on Academic Paper Novelty Assessment
School of Economics and Management, Nanjing University of Science and Technology（南京理工大学经济管理学院）；School of Electronic Engineering and Computer Science, Queen Mary University of London（伦敦玛丽女王大学电子工程与计算机科学学院）；School of Management, Anhui University（安徽大学管理学院）
From Insights to Actions: The Impact of Interpretability and Analysis Research on NLP
Pontificia Universidad Católica de Chile（智利天主教大学）；CENIA（智利国家环境与可持续发展中心）；Mila Quebec AI Institute（米拉魁北克人工智能研究所）；McGill University（麦吉尔大学）；Saarland University（萨尔大学）；Tel Aviv University（特拉维夫大学）
Commitment Checklist: Auditing Author Commitments in Peer Review
AIST, Japan（日本国家信息技术研究所）；Ubiquitous Knowledge Processing Lab (UKP Lab), Department of Computer Science, TU Darmstadt and National Research Center for Applied Cybersecurity ATHENE, Germany（图宾根大学计算机科学系通用知识处理实验室及应用网络安全国家研究中心ATHENE德国）
What Can Natural Language Processing Do for Peer Review?
Technical University of Darmstadt（达姆施塔特工业大学）；Mohamed bin Zayed University of Artificial Intelligence（穆罕默德·本·扎耶德人工智能大学）；Radboud University Nijmegen（奈梅亨拉德堡德大学）；Carnegie Mellon University（卡内基梅隆大学）；Allen Institute for Artificial Intelligence（艾伦人工智能研究所）；Bocconi University（博科尼大学）；The University of Sydney（悉尼大学）；Universität Hamburg（汉堡大学）；University of British Columbia（不列颠哥伦比亚大学）；University of Applied Sciences Darmstadt（达姆施塔特应用科学大学）；Université Paris-Saclay（巴黎萨克雷大学）；Monash University（莫纳什大学）；The Hebrew University of Jerusalem（耶路撒冷希伯来大学）；University of Washington（华盛顿大学）；Georgia Institute of Technology（佐治亚理工学院）；IT University of Copenhagen（哥本哈根信息技术大学）
