AI 能解决同行评审危机吗？关于 LLM 在评估 1000 多篇经济学论文中的表现与偏见的大规模跨模型实验
Can AI Solve the Peer Review Crisis? A Large Scale Cross Model Experiment of LLMs' Performance and Biases in Evaluating over 1000 Economics Papers
AI总结：
本研究通过两项大规模实验评估了四个 LLM 在经济学论文同行评审中的表现与偏见，发现其能有效区分研究质量但会复制人类对作者身份的偏见，为整合 LLM 提升评审效率与公平性提供了实用指导。
摘要：
本研究探讨了大语言模型（LLMs）在不引入系统性偏见的情况下，通过可靠地评估经济学研究质量来增强学术同行评审过程的潜力。我们对四个 LLMs（GPT-4o、Claude 3.5、Gemma 3 和 LLaMA 3.3）进行了首批大规模实验评估之一，包含两项互补实验。在第一项实验中，我们使用非参数 binscatter 和线性回归技术，对从 110 本经济学期刊中抽取的 1,220 篇匿名论文（这些论文被排除在当前 LLMs 的训练数据之外）以及一组 AI 生成的投稿进行了超过 29,000 次评估分析。结果表明，LLMs 能够仅基于文本内容持续区分较高质量和较低质量的研究，产生的质量梯度与既定期刊声望指标高度吻合。Claude 和 Gemma 在捕捉这些梯度方面表现尤为出色，而 GPT 在检测 AI 生成内容方面表现优异。第二项实验包含 8,910 次评估，旨在评估 LLMs 是否在单盲评审中复制类似人类的偏见。通过在 330 篇论文中系统性地改变作者性别、机构隶属关系和学术声望，我们发现 GPT、Gemma 和 LLaMA 对来自顶尖男性作者和精英机构的投稿赋予了显著更高的评分，而同样的论文在匿名呈现时评分较低。这些结果强调了在编辑筛选中部署 LLMs 时排除作者身份信息的重要性。总体而言，我们的发现为将 LLMs 整合到同行评审中以提升经济学研究出版过程的效率、准确性和公平性提供了令人信服的证据和实用指导。
英文摘要：
This study examines the potential of large language models (LLMs) to augment the academic peer review process by reliably evaluating the quality of economics research without introducing systematic bias. We conduct one of the first large-scale experimental assessments of four LLMs (GPT-4o, Claude 3.5, Gemma 3, and LLaMA 3.3) across two complementary experiments. In the first, we use nonparametric binscatter and linear regression techniques to analyze over 29,000 evaluations of 1,220 anonymized papers drawn from 110 economics journals excluded from the training data of current LLMs, along with a set of AI-generated submissions. The results show that LLMs consistently distinguish between higher- and lower-quality research based solely on textual content, producing quality gradients that closely align with established journal prestige measures. Claude and Gemma perform exceptionally well in capturing these gradients, while GPT excels in detecting AI-generated content. The second experiment comprises 8,910 evaluations designed to assess whether LLMs replicate human like biases in single blind reviews. By systematically varying author gender, institutional affiliation, and academic prominence across 330 papers, we find that GPT, Gemma, and LLaMA assign significantly higher ratings to submissions from top male authors and elite institutions relative to the same papers presented anonymously. These results emphasize the importance of excluding author-identifying information when deploying LLMs in editorial screening. Overall, our findings provide compelling evidence and practical guidance for integrating LLMs into peer review to enhance efficiency, improve accuracy, and promote equity in the publication process of economics research.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
