arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2509.04460cs.CLcs.AI
收藏

CoCoNUTS:在AI生成的同行评审检测中聚焦内容同时忽略无信息的文本风格

CoCoNUTS: Concentrating on Content while Neglecting Uninformative Textual Styles for AI-Generated Peer Review Detection

Chinese Information Processing Laboratory, Institute of Software, Chinese Academy of Sciences(中国科学院软件研究所中文信息处理实验室);University of Chinese Academy of Sciences(中国科学院大学)

Yihan Chen, Jiawei Chen, Guozhao Mo, Xuanang Chen, Ben He, Xianpei Han, Le Sun

AI总结:

针对现有AI生成文本检测器依赖风格特征、易误判合规AI润色评审且难识别伪装AI评审的问题,本研究提出内容导向的检测范式,构建含六类人机协作模式的CoCoNUTS基准,并开发多任务学习的CoCoDet检测器以提升检测准确性与鲁棒性。

摘要:

大语言模型(LLMs)在同行评审流程中的日益普及,给学术评估的公平性和可靠性带来了潜在风险。尽管LLMs能为审稿人在语言润色方面提供有价值的帮助,但人们对其被用于生成实质性评审内容的担忧日益加剧。现有的通用AI生成文本检测器易受释义攻击的影响,且难以区分表层语言润色和实质性内容生成,这表明它们主要依赖风格特征。当应用于同行评审场景时,这一局限性会导致系统不公正地怀疑那些经过合规的AI辅助语言优化的评审,同时却无法识别经过伪装、看似人工撰写的AI生成评审。为解决这一问题,我们提出了一种范式转变,即从基于风格的检测转向基于内容的检测。具体而言,我们推出了CoCoNUTS——一个以内容为导向的基准测试集,它基于AI生成同行评审的细粒度数据集构建,涵盖了六种不同的人机协作模式。此外,我们还开发了CoCoDet,这是一款通过多任务学习框架实现的AI评审检测器,旨在更准确、更鲁棒地检测评审内容中AI的参与情况。我们的工作为评估LLMs在同行评审中的使用情况提供了实用基础,并有助于为现实世界的学术应用开发更精准、更公平、更可靠的检测方法。我们的代码和数据将在https://github.com/Y1hanChen/COCONUTS 上公开提供。

英文摘要:

The growing integration of large language models (LLMs) into the peer review process presents potential risks to the fairness and reliability of scholarly evaluation. While LLMs offer valuable assistance for reviewers with language refinement, there is growing concern over their use to generate substantive review content. Existing general AI-generated text detectors are vulnerable to paraphrasing attacks and struggle to distinguish between surface language refinement and substantial content generation, suggesting that they primarily rely on stylistic cues. When applied to peer review, this limitation can result in unfairly suspecting reviews with permissible AI-assisted language enhancement, while failing to catch deceptively humanized AI-generated reviews. To address this, we propose a paradigm shift from style-based to content-based detection. Specifically, we introduce CoCoNUTS, a content-oriented benchmark built upon a fine-grained dataset of AI-generated peer reviews, covering six distinct modes of human-AI collaboration. Furthermore, we develop CoCoDet, an AI review detector via a multi-task learning framework, designed to achieve more accurate and robust detection of AI involvement in review content. Our work offers a practical foundation for evaluating the use of LLMs in peer review, and contributes to the development of more precise, equitable, and reliable detection methods for real-world scholarly applications. Our code and data will be publicly available at https://github.com/Y1hanChen/COCONUTS.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑