Monitoring AI-Modified Content at Scale: A Case Study on the Impact of ChatGPT on AI Conference Peer Reviews
Department of Computer Science, Stanford University(斯坦福大学计算机科学系);Machine Learning Department, NEC Labs America(NEC美国实验室机器学习部门);Department of Biomedical Data Science, Stanford University(斯坦福大学生物医学数据科学系);Department of Electrical Engineering, Stanford University(斯坦福大学电气工程系);Graduate School of Education, Stanford University(斯坦福大学教育研究生院);Department of Sociology, Stanford University(斯坦福大学社会学系);Graduate School of Business, Stanford University(斯坦福大学商学院);Department of Management Science and Engineering, Stanford University(斯坦福大学管理科学与工程系);Department of Computer Science, UC Santa Barbara(加州大学圣芭芭拉分校计算机科学系)
Weixin Liang, Zachary Izzo, Yaohui Zhang, Haley Lepp, Hancheng Cao, Xuandong Zhao, Lingjiao Chen, Haotian Ye, Sheng Liu, Zhi Huang, Daniel A. McFarland, James Y. Zou
We present an approach for estimating the fraction of text in a large corpus which is likely to be substantially modified or produced by a large language model (LLM). Our maximum likelihood model leverages expert-written and AI-generated reference texts to accurately and efficiently examine real-world LLM-use at the corpus level. We apply this approach to a case study of scientific peer review in AI conferences that took place after the release of ChatGPT: ICLR 2024, NeurIPS 2023, CoRL 2023 and EMNLP 2023. Our results suggest that between 6.5% and 16.9% of text submitted as peer reviews to these conferences could have been substantially modified by LLMs, i.e. beyond spell-checking or minor writing updates. The circumstances in which generated text occurs offer insight into user behavior: the estimated fraction of LLM-generated text is higher in reviews which report lower confidence, were submitted close to the deadline, and from reviewers who are less likely to respond to author rebuttals. We also observe corpus-level trends in generated text which may be too subtle to detect at the individual level, and discuss the implications of such trends on peer review. We call for future interdisciplinary work to examine how LLM use is changing our information and knowledge practices.
'Quis custodiet ipsos custodes?' Who will watch the watchmen? On Detecting AI-generated peer-reviews
Indian Institute of Technology Patna(印度理工学院巴特那分校);National Center for Computational Sciences, Oak Ridge National Laboratory(橡树岭国家实验室国家计算科学中心);School of AI and Data Science, IIT Jodhpur(焦特布尔印度理工学院人工智能与数据科学学院)
Commitment Checklist: Auditing Author Commitments in Peer Review
AIST, Japan(日本国家信息技术研究所);Ubiquitous Knowledge Processing Lab (UKP Lab), Department of Computer Science, TU Darmstadt and National Research Center for Applied Cybersecurity ATHENE, Germany(图宾根大学计算机科学系通用知识处理实验室及应用网络安全国家研究中心ATHENE德国)
ReviewerToo: Should AI Join The Program Committee? A Look At The Future of Peer Review
Mila – Quebec AI Institute(魁北克AI研究所);HEC Montréal(蒙特利尔高等商学院);ServiceNow Research(ServiceNow研究);Université de Montréal(蒙特利尔大学);Canada CIFAR Chair(加拿大 CIFAR 席位);Polytechnique Montréal(蒙特利尔理工学院)
本文提出 ReviewerToo 框架,用于研究 AI 辅助同行评审,在 ICLR 2025 数据集上验证其性能接近人类平均水平,并指出 AI 的优势与局限,提出集成指南,以提升评审的一致性和公平性。
How Much Were You Told? Measuring External Information in Peer Reviews
Sorbonne Université(索邦大学);CNRS(法国国家科学研究中心);ISIR(智能系统与机器人研究所);International Laboratory on Learning Systems (ILLS)(国际学习系统实验室);Quebec AI Institute (MILA)(魁北克人工智能研究所);CentraleSupélec(巴黎中央理工-高等电力学院);Université Paris-Saclay(巴黎萨克雷大学)
ChatGPT Chemistry Assistant for Text Mining and Prediction of MOF Synthesis
University of California, Berkeley(加州大学伯克利分校);Kavli Energy Nanoscience Institute, University of California, Berkeley(加州大学伯克利分校卡夫利能源纳米科学研究所);King Abdulaziz City for Science and Technology(阿卜杜勒阿齐兹国王科技城)
Quality of Answers of Generative Large Language Models vs Peer Patients for Interpreting Lab Test Results for Lay Patients: Evaluation Study
School of Information, Florida State University(佛罗里达州立大学信息学院);National Center for Biotechnology Information (NCBI), National Library of Medicine (NLM), National Institutes of Health(美国国立卫生研究院国家医学图书馆国家生物技术信息中心);Morsani College of Medicine, University of South Florida(南佛罗里达大学莫尔萨尼医学院)
Insights from the ICLR Peer Review and Rebuttal Process
LMU Munich(慕尼黑大学);Technical University of Munich(慕尼黑工业大学);Munich Center for Machine Learning(慕尼黑机器学习中心);University of Southern California(南加州大学);Paper Copilot;Cardiff University(卡迪夫大学)
Is Peer Review Really in Decline? Analyzing Review Quality across Venues and Time
Ubiquitous Knowledge Processing Lab (UKP Lab), Department of Computer Science, Technical University of Darmstadt and National Research Center for Applied Cybersecurity ATHENE(技术大学达姆施塔特计算机科学系和国家应用网络安全研究中心ATHENE);Department of Computer Science at Queens College, City University of New York (CUNY)(纽约城市大学皇后学院计算机科学系)
Mapping the Challenges of HCI: An Application and Evaluation of ChatGPT for Mining Insights at Scale
University of Oulu, Center for Applied Computing, Faculty of Information Technology and Electrical Engineering(奥卢大学,应用计算中心,信息科技与电气工程学院);University of Jyväskylä, Software and Communications Engineering, Faculty of Information Technology(耶夫尼耶斯基大学,软件与通信工程,信息科技学院)
What Can Natural Language Processing Do for Peer Review?
Technical University of Darmstadt(达姆施塔特工业大学);Mohamed bin Zayed University of Artificial Intelligence(穆罕默德·本·扎耶德人工智能大学);Radboud University Nijmegen(奈梅亨拉德堡德大学);Carnegie Mellon University(卡内基梅隆大学);Allen Institute for Artificial Intelligence(艾伦人工智能研究所);Bocconi University(博科尼大学);The University of Sydney(悉尼大学);Universität Hamburg(汉堡大学);University of British Columbia(不列颠哥伦比亚大学);University of Applied Sciences Darmstadt(达姆施塔特应用科学大学);Université Paris-Saclay(巴黎萨克雷大学);Monash University(莫纳什大学);The Hebrew University of Jerusalem(耶路撒冷希伯来大学);University of Washington(华盛顿大学);Georgia Institute of Technology(佐治亚理工学院);IT University of Copenhagen(哥本哈根信息技术大学)
When Reviews Disagree: Fine-Grained Contradiction Analysis in Scientific Peer Reviews
Department of Computer Science and Engineering, Indian Institute of Technology Patna, India(印度理工学院帕纳瓦分校计算机科学与工程系);School of Computer Engineering, KIIT Deemed to be University, Bhubaneswar, India(比哈尔邦布尔萨大学计算机工程学院)