arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2405.06563cs.CL
收藏

自然语言处理能为同行评审做什么?

What Can Natural Language Processing Do for Peer Review?

Technical University of Darmstadt(达姆施塔特工业大学);Mohamed bin Zayed University of Artificial Intelligence(穆罕默德·本·扎耶德人工智能大学);Radboud University Nijmegen(奈梅亨拉德堡德大学);Carnegie Mellon University(卡内基梅隆大学);Allen Institute for Artificial Intelligence(艾伦人工智能研究所);Bocconi University(博科尼大学);The University of Sydney(悉尼大学);Universität Hamburg(汉堡大学);University of British Columbia(不列颠哥伦比亚大学);University of Applied Sciences Darmstadt(达姆施塔特应用科学大学);Université Paris-Saclay(巴黎萨克雷大学);Monash University(莫纳什大学);The Hebrew University of Jerusalem(耶路撒冷希伯来大学);University of Washington(华盛顿大学);Georgia Institute of Technology(佐治亚理工学院);IT University of Copenhagen(哥本哈根信息技术大学)
Ilia Kuznetsov, Osama Mohammed Afzal, Koen Dercksen, Nils Dycke, Alexander Goldberg, Tom Hope, Dirk Hovy, Jonathan K. Kummerfeld, Anne Lauscher, Kevin Leyton-Br… 展开作者

Ilia Kuznetsov, Osama Mohammed Afzal, Koen Dercksen, Nils Dycke, Alexander Goldberg, Tom Hope, Dirk Hovy, Jonathan K. Kummerfeld, Anne Lauscher, Kevin Leyton-Brown, Sheng Lu, Mausam, Margot Mieskes, Aurélie Névéol, Danish Pruthi, Lizhen Qu, Roy Schwartz, Noah A. Smith, Thamar Solorio, Jingyan Wang, Xiaodan Zhu, Anna Rogers, Nihar B. Shah, Iryna Gurevych

AI总结:

本文系统梳理同行评审全流程中NLP可辅助的环节、挑战与伦理边界,并汇总相关数据集,为机器辅助科学质量控制研究制定议程。

摘要:

每年产生的科学论文数量正在快速增长。对这些论文进行质量控制,对科学家乃至最终的公共利益都至关重要。在现代科学中,这一过程在很大程度上被委托给同行评审——这是一种分布式流程,每篇投稿都由该领域的若干独立专家进行评估。同行评审被广泛使用,但它既困难、耗时,又容易出错。由于同行评审涉及的产物——手稿、评审意见、讨论——在很大程度上都以文本为基础,自然语言处理(NLP)在改进评审方面具有巨大潜力。随着大型语言模型(LLM)的出现,NLP已能够为许多新任务提供辅助,关于机器辅助同行评审的讨论也正在加速。然而,究竟哪里需要帮助,NLP能在哪里提供帮助,又应当在哪些地方置身事外?本文的目标是为未来利用NLP辅助同行评审的工作奠定基础。我们将同行评审作为一个一般过程加以讨论,并以人工智能会议的评审为例。我们详细梳理了从手稿投稿到最终可印刷版本修订的每一个流程步骤,并结合现有工作讨论NLP辅助所面临的相关挑战与机遇。随后,我们转向NLP用于同行评审这一整体领域中的重大挑战,包括数据获取与许可、操作化与实验,以及伦理问题。为帮助凝聚社区力量,我们创建了一个配套代码仓库,汇总与同行评审相关的关键数据集。最后,我们向科学界、NLP与人工智能研究者、政策制定者以及资助机构发出详细的行动呼吁,以推动NLP用于同行评审的研究。我们希望本文有助于在NLP社区及更广泛范围内,确定人工智能时代机器辅助科学质量控制研究的议程。

英文摘要:

The number of scientific articles produced every year is growing rapidly. Providing quality control over them is crucial for scientists and, ultimately, for the public good. In modern science, this process is largely delegated to peer review -- a distributed procedure in which each submission is evaluated by several independent experts in the field. Peer review is widely used, yet it is hard, time-consuming, and prone to error. Since the artifacts involved in peer review -- manuscripts, reviews, discussions -- are largely text-based, Natural Language Processing has great potential to improve reviewing. As the emergence of large language models (LLMs) has enabled NLP assistance for many new tasks, the discussion on machine-assisted peer review is picking up the pace. Yet, where exactly is help needed, where can NLP help, and where should it stand aside? The goal of our paper is to provide a foundation for the future efforts in NLP for peer-reviewing assistance. We discuss peer review as a general process, exemplified by reviewing at AI conferences. We detail each step of the process from manuscript submission to camera-ready revision, and discuss the associated challenges and opportunities for NLP assistance, illustrated by existing work. We then turn to the big challenges in NLP for peer review as a whole, including data acquisition and licensing, operationalization and experimentation, and ethical issues. To help consolidate community efforts, we create a companion repository that aggregates key datasets pertaining to peer review. Finally, we issue a detailed call for action for the scientific community, NLP and AI researchers, policymakers, and funding bodies to help bring the research in NLP for peer review forward. We hope that our work will help set the agenda for research in machine-assisted scientific quality control in the age of AI, within the NLP community and beyond.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑