arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2605.21713cs.CL
收藏

Sem-Detect:基于语义层面的AI生成同行评审检测

Sem-Detect: Semantic Level Detection of AI Generated Peer-Reviews

Language Technologies Institute, Carnegie Mellon University(卡内基梅隆大学语言技术研究所)

André V. Duarte, Brian Tufts, Aditya Oke, Fei Fang, Arlindo L. Oliveira, Lei Li

AI总结:

本文提出Sem-Detect方法,通过结合文本特征和语义分析,区分AI生成与人类撰写的同行评审,实验表明其在二分类和三分类场景下均表现出色,准确率显著提升。

摘要:

如何区分同行评审是人类还是AI生成?我们主张在该设置中,不应仅根据文本特征来确定作者身份,还应考虑评审所表达的观点、判断和主张。为此,我们提出了Sem-Detect,一种用于同行评审的作者身份检测方法,通过结合文本特征和基于声明的语义分析来实现这一原则。Sem-Detect通过将目标评审与同一篇论文的多个AI生成评审进行比较,利用观察到的不同AI模型倾向于收敛到相似点,而人类评审者引入更多独特和多样化的观点这一现象。因此,Sem-Detect能够区分完全由AI生成的评审与真实的由人类撰写的评审,包括那些经过LLM优化但仍反映人类判断的评审。在包含超过20,000篇同行评审的ICLR和NeurIPS会议数据集中,Sem-Detect在二分类设置中将TPR@0.1% FPR比最强基线提高了25.5%。此外,在三分类场景中,我们实证表明LLM优化保留了人类评审的语义信号,这些信号仍与完全由AI生成的文本模式区分开来;因此,少于3.5%的LLM优化后的评审被错误分类为AI生成。

英文摘要:

How can we distinguish whether a peer review was written by a human or generated by an AI model? We argue that, in this setting, authorship should not be attributed solely from the textual features of a review, but also from the ideas, judgments, and claims it expresses. To this end, we propose Sem-Detect, an authorship detection method for peer reviews that operationalizes this principle by combining textual features with claim-level semantic analysis. Sem-Detect compares a target review against multiple AI-generated reviews of the same paper, leveraging the observation that different AI models tend to converge on similar points, while human reviewers introduce more unique and diverse ones. As a result, Sem-Detect is able to distinguish fully AI reviews from authentic human-written ones, including those that have been refined using an LLM but still reflect human judgment. Across a dataset of over 20,000 peer reviews from ICLR and NeurIPS conferences, Sem-Detect improves over the strongest baseline by 25.5% in TPR@0.1% FPR in the binary setting. Moreover, in the three-class scenario, we empirically show that LLM refinement preserves the semantic signals of human reviews, which remain distinct from the patterns exhibited by fully AI-generated text; as a result, fewer than 3.5% of LLM-refined human reviews are misclassified as AI-generated.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑