arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2606.22166cs.DLcs.AIcs.LG
收藏

反驳会改变同行评审分数,但初始评审结构限制了变化幅度

Rebuttals Move Peer-Review Scores, but Initial-Review Structure Bounds the Movement

Data Analytics Lab, Vrije Universiteit Brussel(自由大学布鲁塞尔数据分析实验室);imec-SMIT, Vrije Universiteit Brussel(imec-SMIT,自由大学布鲁塞尔);School of Engineering and Applied Sciences, Harvard University(哈佛大学工程与应用科学学院)

Mathieu Louis, Tibo Vanleke, Vincent Ginis, Andres Algaba

AI总结:

利用ICLR 2024-2025的73,000条评审轨迹,结合LLM分析,发现反驳能改变分数,但初始评审结构已预测大部分变化,且可测量的交换信号多为反驳失败模式。

摘要:

作者反驳是同行评审中主要的投稿后窗口,但其对评审分数的影响仍难以衡量,因为分数更新混合了反驳内容、初始分数位置、论文层面共识、评审者信心和讨论动态。我们使用ICLR 2024-2025的73,000条评审轨迹,其中包含外部存档的反驳前和反驳后分数,并仅将LLM用作测量工具。Gemini Flash 3.0从去除分数的评审文本中预测隐含的反驳前分数。由此产生的文本-分数偏移预测了后续变化,当文本读数低于给定分数时,分数上升率从8.3%增加到当文本读数高于分数时的31.9%。Claude Opus 4.6归纳出,并由结果盲化的Gemini Flash 3.0验证,一个包含44个特征的已解决评审者-作者交流分类体系,其中23个特征在Bonferroni校正下跨模型和保留年份复制。在反驳参与的基准测试(n=6,705)中,初始评审结构已经预测了大部分分数变化(AUC=0.747,最小AUC=0.696),而添加已解决交流将AUC提高到0.804。反驳可以改变分数,但可测量的变化受限于初始评审结构,且稳健的交流信号大多是反驳失败模式。

英文摘要:

Author rebuttals are the main post-submission window in peer review, but their effect on reviewer scores remains hard to measure because score updates mix rebuttal content with initial score position, paper-level consensus, reviewer confidence, and discussion dynamics. We study ICLR 2024-2025 using 73,000 reviewer trajectories with externally archived pre- and post-rebuttal scores, and use LLMs only as measurement instruments. Gemini Flash 3.0 predicts implied pre-rebuttal scores from score-stripped review text. The resulting text-score offset predicts later movement, with score-increase rates rising from 8.3% when text reads below the assigned score to 31.9% when it reads above. Claude Opus 4.6 induces, and outcome-blinded Gemini Flash 3.0 validates, a 44-feature taxonomy of resolved reviewer-author exchanges, where 23 features replicate across model and held-out year under Bonferroni correction. In the rebuttal-engaged benchmark (n=6,705), initial-review structure already predicts much score movement (AUC=0.747, minimal AUC=0.696), while adding the resolved exchange raises AUC to 0.804. Rebuttals can move scores, but measurable movement is bounded by initial-review structure, and robust exchange signals are mostly rebuttal failure modes.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑