'Quis custodiet ipsos custodes?' 谁来监督监督者？关于检测AI生成的同行评审
'Quis custodiet ipsos custodes?' Who will watch the watchmen? On Detecting AI-generated peer-reviews
AI总结：
本文提出TF和RR两种模型检测同行评审是否由ChatGPT生成，并验证其优于其他AI文本检测器，同时提出防御策略以减少释义攻击的影响。
摘要：
同行评审过程的完整性对于维持学术界的科学严谨性和信任至关重要。随着大型语言模型（LLMs）如ChatGPT在学术写作中的使用稳步增加，人们越来越担心AI生成的文本可能会损害科学出版，包括同行评审。以往的工作主要集中在通用的AI生成文本检测，或者提出了一种估计同行评审中可能由AI生成分数的方法。我们这里的重点是通过协助编辑或主席确定评审是否由ChatGPT编写来解决现实世界的问题。为此，我们引入了词频（TF）模型，该模型假设AI经常重复标记，以及评审重生成（RR）模型，该模型基于ChatGPT在重新提示时生成相似输出的想法。我们对这些检测器进行了针对标记攻击和释义的压力测试。最后，我们提出了一种有效的防御策略，以减少释义对我们模型的影响。我们的研究结果表明，我们提出的两种方法均优于其他AI文本检测器。我们的RR模型更鲁棒，尽管在没有攻击的情况下，我们的TF模型表现优于RR模型。我们公开了我们的代码、数据集和模型。
英文摘要：
The integrity of the peer-review process is vital for maintaining scientific rigor and trust within the academic community. With the steady increase in the usage of large language models (LLMs) like ChatGPT in academic writing, there is a growing concern that AI-generated texts could compromise scientific publishing, including peer-reviews. Previous works have focused on generic AI-generated text detection or have presented an approach for estimating the fraction of peer-reviews that can be AI-generated. Our focus here is to solve a real-world problem by assisting the editor or chair in determining whether a review is written by ChatGPT or not. To address this, we introduce the Term Frequency (TF) model, which posits that AI often repeats tokens, and the Review Regeneration (RR) model, which is based on the idea that ChatGPT generates similar outputs upon re-prompting. We stress test these detectors against token attack and paraphrasing. Finally, we propose an effective defensive strategy to reduce the effect of paraphrasing on our models. Our findings suggest both our proposed methods perform better than the other AI text detectors. Our RR model is more robust, although our TF model performs better than the RR model without any attacks. We make our code, dataset, and model public.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
