What Can Natural Language Processing Do for Peer Review?
Technical University of Darmstadt(达姆施塔特工业大学);Mohamed bin Zayed University of Artificial Intelligence(穆罕默德·本·扎耶德人工智能大学);Radboud University Nijmegen(奈梅亨拉德堡德大学);Carnegie Mellon University(卡内基梅隆大学);Allen Institute for Artificial Intelligence(艾伦人工智能研究所);Bocconi University(博科尼大学);The University of Sydney(悉尼大学);Universität Hamburg(汉堡大学);University of British Columbia(不列颠哥伦比亚大学);University of Applied Sciences Darmstadt(达姆施塔特应用科学大学);Université Paris-Saclay(巴黎萨克雷大学);Monash University(莫纳什大学);The Hebrew University of Jerusalem(耶路撒冷希伯来大学);University of Washington(华盛顿大学);Georgia Institute of Technology(佐治亚理工学院);IT University of Copenhagen(哥本哈根信息技术大学)
Ilia Kuznetsov, Osama Mohammed Afzal, Koen Dercksen, Nils Dycke, Alexander Goldberg, Tom Hope, Dirk Hovy, Jonathan K. Kummerfeld, Anne Lauscher, Kevin Leyton-Br… 展开作者
Ilia Kuznetsov, Osama Mohammed Afzal, Koen Dercksen, Nils Dycke, Alexander Goldberg, Tom Hope, Dirk Hovy, Jonathan K. Kummerfeld, Anne Lauscher, Kevin Leyton-Brown, Sheng Lu, Mausam, Margot Mieskes, Aurélie Névéol, Danish Pruthi, Lizhen Qu, Roy Schwartz, Noah A. Smith, Thamar Solorio, Jingyan Wang, Xiaodan Zhu, Anna Rogers, Nihar B. Shah, Iryna Gurevych
The number of scientific articles produced every year is growing rapidly. Providing quality control over them is crucial for scientists and, ultimately, for the public good. In modern science, this process is largely delegated to peer review -- a distributed procedure in which each submission is evaluated by several independent experts in the field. Peer review is widely used, yet it is hard, time-consuming, and prone to error. Since the artifacts involved in peer review -- manuscripts, reviews, discussions -- are largely text-based, Natural Language Processing has great potential to improve reviewing. As the emergence of large language models (LLMs) has enabled NLP assistance for many new tasks, the discussion on machine-assisted peer review is picking up the pace. Yet, where exactly is help needed, where can NLP help, and where should it stand aside? The goal of our paper is to provide a foundation for the future efforts in NLP for peer-reviewing assistance. We discuss peer review as a general process, exemplified by reviewing at AI conferences. We detail each step of the process from manuscript submission to camera-ready revision, and discuss the associated challenges and opportunities for NLP assistance, illustrated by existing work. We then turn to the big challenges in NLP for peer review as a whole, including data acquisition and licensing, operationalization and experimentation, and ethical issues. To help consolidate community efforts, we create a companion repository that aggregates key datasets pertaining to peer review. Finally, we issue a detailed call for action for the scientific community, NLP and AI researchers, policymakers, and funding bodies to help bring the research in NLP for peer review forward. We hope that our work will help set the agenda for research in machine-assisted scientific quality control in the age of AI, within the NLP community and beyond.
AgentReview: Exploring Peer Review Dynamics with LLM Agents
Georgia Institute of Technology(佐治亚理工学院);University of Science and Technology of China(中国科学技术大学);Carnegie Mellon University(卡内基梅隆大学);University of California, Santa Barbara(加州大学圣巴巴拉分校);University of California, Los Angeles(加州大学洛杉矶分校);William & Mary(威廉与玛丽学院)
ReviewerToo: Should AI Join The Program Committee? A Look At The Future of Peer Review
Mila – Quebec AI Institute(魁北克AI研究所);HEC Montréal(蒙特利尔高等商学院);ServiceNow Research(ServiceNow研究);Université de Montréal(蒙特利尔大学);Canada CIFAR Chair(加拿大 CIFAR 席位);Polytechnique Montréal(蒙特利尔理工学院)
本文提出 ReviewerToo 框架,用于研究 AI 辅助同行评审,在 ICLR 2025 数据集上验证其性能接近人类平均水平,并指出 AI 的优势与局限,提出集成指南,以提升评审的一致性和公平性。
CoCoNUTS: Concentrating on Content while Neglecting Uninformative Textual Styles for AI-Generated Peer Review Detection
Chinese Information Processing Laboratory, Institute of Software, Chinese Academy of Sciences(中国科学院软件研究所中文信息处理实验室);University of Chinese Academy of Sciences(中国科学院大学)
NLPeer: A Unified Resource for the Computational Study of Peer Review
Ubiquitous Knowledge Processing Lab (UKP Lab)(普适知识处理实验室(UKP实验室));Department of Computer Science(计算机科学系);Hessian Center for AI (hessian.AI)(黑森人工智能中心(hessian.AI));Technical University of Darmstadt(达姆施塔特工业大学)
COMPARE: A Taxonomy and Dataset of Comparison Discussions in Peer Reviews
Engg. Indian Institute of Technology, Gandhinagar Gujarat, India singh\;Engg. Indian Institute of Technology, Gandhinagar Gujarat, India;Engg. Indian Institute of Technology, Kharagpur West Bengal, India
When Reviews Disagree: Fine-Grained Contradiction Analysis in Scientific Peer Reviews
Department of Computer Science and Engineering, Indian Institute of Technology Patna, India(印度理工学院帕纳瓦分校计算机科学与工程系);School of Computer Engineering, KIIT Deemed to be University, Bhubaneswar, India(比哈尔邦布尔萨大学计算机工程学院)
MOPRD: A multidisciplinary open peer review dataset
School of Informatics, Xiamen University(厦门大学信息学院);Key Laboratory of Digital Protection and Intelligent Processing of Intangible Cultural Heritage of Fujian and Taiwan, Ministry of Culture and Tourism(文化和旅游部闽台非物质文化遗产数字化保护与智能处理重点实验室);College of Foreign Languages and Cultures, Xiamen University(厦门大学外文学院)
FMMD: A multimodal multidisciplinary dataset of open peer reviews from F1000Research
School of Computer Science and Engineering, Guangzhou Institute of Science and Technology(计算机科学与工程学院,广州科学与技术研究所);College of Foreign Languages and Cultures, Xiamen University(外语学院,厦门大学);Science and Education Evaluation Lab, Guangzhou Institute of Science and Technology(科学与教育评估实验室,广州科学与技术研究所)