AI评审抽奖:广泛的AI辅助同行评审推高论文分数与接收率
The AI Review Lottery: Widespread AI-Assisted Peer Reviews Boost Paper Scores and Acceptance Rates
AI总结:
本文以ICLR 2024为对象开展准实验研究,发现至少15.8%的评审由AI辅助撰写,且AI辅助评审会显著推高论文分数和接收率,揭示了AI对同行评审公平性的实质影响。
摘要:
期刊和会议担忧由人工智能(AI),尤其是大语言模型(LLMs)辅助的同行评审可能对现代科学基石的同行评审系统的有效性和公平性产生负面影响。在本工作中,我们以2024年国际学习表征会议(ICLR)——一个大型且享有盛誉的机器学习会议——为背景,通过一项准实验研究来回应这一担忧,考察AI辅助同行评审的普遍性和影响。我们的贡献有三方面。首先,我们使用GPTZero LLM检测器获得了ICLR 2024中AI辅助评审普遍性的下界,估计至少有15.8%的评审是在AI辅助下撰写的。其次,我们估计了AI辅助评审对投稿分数的影响。考虑针对同一篇论文给出不同分数的评审对,我们发现在53.4%的评审对中,AI辅助评审给出的分数高于人类评审(p = 0.002;在给出更高分数的概率上的相对差异为+14.4%,有利于AI辅助评审)。第三,我们评估了收到AI辅助同行评审对投稿被接收的影响。在一项匹配研究中,接近接收阈值且收到AI辅助同行评审的投稿比未收到此类评审的投稿被接收的可能性高出4.9个百分点(p = 0.024)。总体而言,我们表明AI辅助评审对同行评审过程具有重要影响,并就当前趋势的未来影响展开了讨论。
英文摘要:
Journals and conferences worry that peer reviews assisted by artificial intelligence (AI), in particular, large language models (LLMs), may negatively influence the validity and fairness of the peer-review system, a cornerstone of modern science. In this work, we address this concern with a quasi-experimental study of the prevalence and impact of AI-assisted peer reviews in the context of the 2024 International Conference on Learning Representations (ICLR), a large and prestigious machine-learning conference. Our contributions are threefold. Firstly, we obtain a lower bound for the prevalence of AI-assisted reviews at ICLR 2024 using the GPTZero LLM detector, estimating that at least $15.8\%$ of reviews were written with AI assistance. Secondly, we estimate the impact of AI-assisted reviews on submission scores. Considering pairs of reviews with different scores assigned to the same paper, we find that in $53.4\%$ of pairs the AI-assisted review scores higher than the human review ($p = 0.002$; relative difference in probability of scoring higher: $+14.4\%$ in favor of AI-assisted reviews). Thirdly, we assess the impact of receiving an AI-assisted peer review on submission acceptance. In a matched study, submissions near the acceptance threshold that received an AI-assisted peer review were $4.9$ percentage points ($p = 0.024$) more likely to be accepted than submissions that did not. Overall, we show that AI-assisted reviews are consequential to the peer-review process and offer a discussion on future implications of current trends
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。