作者反驳是同行评审中主要的投稿后窗口,但其对评审分数的影响仍难以衡量,因为分数更新混合了反驳内容、初始分数位置、论文层面共识、评审者信心和讨论动态。我们使用ICLR 2024-2025的73,000条评审轨迹,其中包含外部存档的反驳前和反驳后分数,并仅将LLM用作测量工具。Gemini Flash 3.0从去除分数的评审文本中预测隐含的反驳前分数。由此产生的文本-分数偏移预测了后续变化,当文本读数低于给定分数时,分数上升率从8.3%增加到当文本读数高于分数时的31.9%。Claude Opus 4.6归纳出,并由结果盲化的Gemini Flash 3.0验证,一个包含44个特征的已解决评审者-作者交流分类体系,其中23个特征在Bonferroni校正下跨模型和保留年份复制。在反驳参与的基准测试(n=6,705)中,初始评审结构已经预测了大部分分数变化(AUC=0.747,最小AUC=0.696),而添加已解决交流将AUC提高到0.804。反驳可以改变分数,但可测量的变化受限于初始评审结构,且稳健的交流信号大多是反驳失败模式。
英文摘要:
Author rebuttals are the main post-submission window in peer review, but their effect on reviewer scores remains hard to measure because score updates mix rebuttal content with initial score position, paper-level consensus, reviewer confidence, and discussion dynamics. We study ICLR 2024-2025 using 73,000 reviewer trajectories with externally archived pre- and post-rebuttal scores, and use LLMs only as measurement instruments. Gemini Flash 3.0 predicts implied pre-rebuttal scores from score-stripped review text. The resulting text-score offset predicts later movement, with score-increase rates rising from 8.3% when text reads below the assigned score to 31.9% when it reads above. Claude Opus 4.6 induces, and outcome-blinded Gemini Flash 3.0 validates, a 44-feature taxonomy of resolved reviewer-author exchanges, where 23 features replicate across model and held-out year under Bonferroni correction. In the rebuttal-engaged benchmark (n=6,705), initial-review structure already predicts much score movement (AUC=0.747, minimal AUC=0.696), while adding the resolved exchange raises AUC to 0.804. Rebuttals can move scores, but measurable movement is bounded by initial-review structure, and robust exchange signals are mostly rebuttal failure modes.
Insights from the ICLR Peer Review and Rebuttal Process
LMU Munich(慕尼黑大学);Technical University of Munich(慕尼黑工业大学);Munich Center for Machine Learning(慕尼黑机器学习中心);University of Southern California(南加州大学);Paper Copilot;Cardiff University(卡迪夫大学)
Monitoring AI-Modified Content at Scale: A Case Study on the Impact of ChatGPT on AI Conference Peer Reviews
Department of Computer Science, Stanford University(斯坦福大学计算机科学系);Machine Learning Department, NEC Labs America(NEC美国实验室机器学习部门);Department of Biomedical Data Science, Stanford University(斯坦福大学生物医学数据科学系);Department of Electrical Engineering, Stanford University(斯坦福大学电气工程系);Graduate School of Education, Stanford University(斯坦福大学教育研究生院);Department of Sociology, Stanford University(斯坦福大学社会学系);Graduate School of Business, Stanford University(斯坦福大学商学院);Department of Management Science and Engineering, Stanford University(斯坦福大学管理科学与工程系);Department of Computer Science, UC Santa Barbara(加州大学圣芭芭拉分校计算机科学系)
MOPRD: A multidisciplinary open peer review dataset
School of Informatics, Xiamen University(厦门大学信息学院);Key Laboratory of Digital Protection and Intelligent Processing of Intangible Cultural Heritage of Fujian and Taiwan, Ministry of Culture and Tourism(文化和旅游部闽台非物质文化遗产数字化保护与智能处理重点实验室);College of Foreign Languages and Cultures, Xiamen University(厦门大学外文学院)
Can LLM feedback enhance review quality? A randomized study of 20K reviews at ICLR 2025
Stanford University(斯坦福大学);Georgia Institute of Technology(佐治亚理工学院);University of California, Los Angeles(加利福尼亚大学洛杉矶分校);Google Research(谷歌研究院);University of California, San Diego(加利福尼亚大学圣迭戈分校);Columbia University(哥伦比亚大学)
Commitment Checklist: Auditing Author Commitments in Peer Review
AIST, Japan(日本国家信息技术研究所);Ubiquitous Knowledge Processing Lab (UKP Lab), Department of Computer Science, TU Darmstadt and National Research Center for Applied Cybersecurity ATHENE, Germany(图宾根大学计算机科学系通用知识处理实验室及应用网络安全国家研究中心ATHENE德国)