arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2610.19339cs.AI
收藏

我们能相信AI同行评审吗?

Can We Believe AI Peer Review?

University of Auckland(奥克兰大学)

Zongrui Yang, Haoyuan Li, Zhongsheng Wang, Gillian Dobbie, Jiamou Liu

AI总结:

本研究以ICLR 2025与2026年200篇论文的约14000条AI评审为对象,探究AI同行评审的可靠性等问题,发现重复评审有波动,精细评分协议可降方差但会改平均分,修改成果分配依赖评审者。

摘要:

随着大语言模型在科学评审和稿件修改中被广泛应用,AI评分正日益作为评估论文质量和指导修改的反馈依据。但单个AI评分是否可靠反映评审者判断、基于该评分的修改成果是否获其他评审认可,仍待研究。本研究使用ICLR 2025与2026年的200篇论文及约14000条AI评审,探究评分可靠性、评分分布估计及修改成果的跨模型一致性。结果显示,相同条件下重复评审仍有波动;更精细的评分协议可降低评分方差,但会系统性改变平均评分;稿件修改的成果分配高度依赖评审者。

英文摘要:

As large language models become widely used in scientific review and manuscript revision, AI scores are increasingly serving as feedback for assessing paper quality and guiding edits. Yet whether a single AI score reliably reflects the reviewer's judgment, and whether revision gains based on that score are endorsed by other reviewers, remain understudied. Using 200 papers from ICLR 2025 and 2026 and approximately 14,000 AI reviews, we study score reliability, score-distribution estimation, and cross-model agreement on revision gains. Repeated reviews fluctuate even under identical conditions. Finer-grained scoring protocols reduce score variance but also systematically shift mean scores, while the gains assigned to manuscript revisions depend strongly on the reviewer.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑