arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2406.12708cs.CL
收藏

AgentReview: 探索基于LLM代理的同行评审动态

AgentReview: Exploring Peer Review Dynamics with LLM Agents

Georgia Institute of Technology(佐治亚理工学院);University of Science and Technology of China(中国科学技术大学);Carnegie Mellon University(卡内基梅隆大学);University of California, Santa Barbara(加州大学圣巴巴拉分校);University of California, Los Angeles(加州大学洛杉矶分校);William & Mary(威廉与玛丽学院)

Yiqiao Jin, Qinlin Zhao, Yiyang Wang, Hao Chen, Kaijie Zhu, Yijia Xiao, Jindong Wang

AI总结:

本文提出AgentReview框架,通过LLM模拟同行评审过程,揭示评审者偏见导致论文决策37.1%的变异,结合社会学理论提升评审机制设计。

摘要:

同行评审是科学出版物完整性和发展的基石。传统同行评审分析方法往往依赖现有评审数据的探索和统计,无法充分处理过程的多变量性,未考虑潜在变量,且因数据敏感性受到隐私问题的限制。我们引入AgentReview,首个基于大语言模型(LLM)的同行评审模拟框架,有效解耦了多个潜在因素的影响并解决了隐私问题。我们的研究揭示了重要见解,包括由于评审者偏见导致的论文决策显著变异(37.1%),并支持社会影响理论、利他疲劳和权威偏见等社会学理论。我们相信,本研究可为改进同行评审机制设计提供有价值的见解。我们的代码可在https://github.com/Ahren09/AgentReview上获得。

英文摘要:

Peer review is fundamental to the integrity and advancement of scientific publication. Traditional methods of peer review analyses often rely on exploration and statistics of existing peer review data, which do not adequately address the multivariate nature of the process, account for the latent variables, and are further constrained by privacy concerns due to the sensitive nature of the data. We introduce AgentReview, the first large language model (LLM) based peer review simulation framework, which effectively disentangles the impacts of multiple latent factors and addresses the privacy issue. Our study reveals significant insights, including a notable 37.1% variation in paper decisions due to reviewers' biases, supported by sociological theories such as the social influence theory, altruism fatigue, and authority bias. We believe that this study could offer valuable insights to improve the design of peer review mechanisms. Our code is available at https://github.com/Ahren09/AgentReview.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑