arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2410.03019cs.CLcs.AI
收藏

你的论文正在被 LLM 评审吗?调查同行评审中 AI 文本的可检测性

Is Your Paper Being Reviewed by an LLM? Investigating AI Text Detectability in Peer Review

Intel Labs(英特尔实验室)

Sungduk Yu, Man Luo, Avinash Madasu, Vasudev Lal, Phillip Howard

AI总结:

本文研究同行评审中 LLM 生成文本的检测问题,评估现有检测器并提出新方法,以在低假阳性率下更有效识别 GPT-4o 撰写的评审。

摘要:

同行评审是确保已发表科学研究完整性的关键过程。对这一过程的信任建立在如下假设之上:相关领域的专家会认真考虑投稿稿件的优点。随着大型语言模型(LLM)语言能力近期的快速进步,同行评审过程面临一种新的潜在风险,即不负责任的评审者会依赖 LLM 来完成通常十分耗时的论文评审工作。在本研究中,我们调查了现有 AI 文本检测算法区分人类撰写的同行评审与不同最先进 LLM 所写评审的能力。我们的分析表明,现有方法无法识别许多由 GPT-4o 撰写的评审,同时还会产生大量假阳性分类。为解决这一不足,我们提出了一种新的检测方法,该方法在较低假阳性分类水平下识别 GPT-4o 撰写的同行评审方面优于现有方法。我们的工作揭示了在单条评审层面准确识别 AI 生成文本的困难,凸显了迫切需要新的工具和方法,以检测生成式 AI 的此类不道德应用。

英文摘要:

Peer review is a critical process for ensuring the integrity of published scientific research. Confidence in this process is predicated on the assumption that experts in the relevant domain give careful consideration to the merits of manuscripts which are submitted for publication. With the recent rapid advancements in the linguistic capabilities of large language models (LLMs), a new potential risk to the peer review process is that negligent reviewers will rely on LLMs to perform the often time consuming process of reviewing a paper. In this study, we investigate the ability of existing AI text detection algorithms to distinguish between peer reviews written by humans and different state-of-the-art LLMs. Our analysis shows that existing approaches fail to identify many GPT-4o written reviews without also producing a high number of false positive classifications. To address this deficiency, we propose a new detection approach which surpasses existing methods in the identification of GPT-4o written peer reviews at low levels of false positive classifications. Our work reveals the difficulty of accurately identifying AI-generated text at the individual review level, highlighting the urgent need for new tools and methods to detect this type of unethical application of generative AI.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑