How can we distinguish whether a peer review was written by a human or generated by an AI model? We argue that, in this setting, authorship should not be attributed solely from the textual features of a review, but also from the ideas, judgments, and claims it expresses. To this end, we propose Sem-Detect, an authorship detection method for peer reviews that operationalizes this principle by combining textual features with claim-level semantic analysis. Sem-Detect compares a target review against multiple AI-generated reviews of the same paper, leveraging the observation that different AI models tend to converge on similar points, while human reviewers introduce more unique and diverse ones. As a result, Sem-Detect is able to distinguish fully AI reviews from authentic human-written ones, including those that have been refined using an LLM but still reflect human judgment. Across a dataset of over 20,000 peer reviews from ICLR and NeurIPS conferences, Sem-Detect improves over the strongest baseline by 25.5% in TPR@0.1% FPR in the binary setting. Moreover, in the three-class scenario, we empirically show that LLM refinement preserves the semantic signals of human reviews, which remain distinct from the patterns exhibited by fully AI-generated text; as a result, fewer than 3.5% of LLM-refined human reviews are misclassified as AI-generated.
CoCoNUTS: Concentrating on Content while Neglecting Uninformative Textual Styles for AI-Generated Peer Review Detection
Chinese Information Processing Laboratory, Institute of Software, Chinese Academy of Sciences(中国科学院软件研究所中文信息处理实验室);University of Chinese Academy of Sciences(中国科学院大学)
LazyReview A Dataset for Uncovering Lazy Thinking in NLP Peer Reviews
Ubiquitous Knowledge Processing Lab, Department of Computer Science and Hessian Center for AI (hessian.AI), Technical University of Darmstadt(德意志联邦共和国达姆施塔特技术大学);School of Computing Technologies, Royal Melbourne Institute of Technology, Australia(澳大利亚皇家墨尔本理工学院);Data Science Group, University of Hamburg(汉堡大学);Department of Data Science & AI, Monash University, Australia(澳大利亚墨尔本大学)
'Quis custodiet ipsos custodes?' Who will watch the watchmen? On Detecting AI-generated peer-reviews
Indian Institute of Technology Patna(印度理工学院巴特那分校);National Center for Computational Sciences, Oak Ridge National Laboratory(橡树岭国家实验室国家计算科学中心);School of AI and Data Science, IIT Jodhpur(焦特布尔印度理工学院人工智能与数据科学学院)
Large Language Models Penetration in Scholarly Writing and Peer Review
The Chinese University of Hong Kong, Shenzhen(香港中文大学(深圳));University of Electronic Science and Technology of China(电子科技大学);Wuhan University of Technology(武汉理工大学);University of Copenhagen(哥本哈根大学);Shenzhen Research Institute of Big Data(深圳大数据研究院)
Commitment Checklist: Auditing Author Commitments in Peer Review
AIST, Japan(日本国家信息技术研究所);Ubiquitous Knowledge Processing Lab (UKP Lab), Department of Computer Science, TU Darmstadt and National Research Center for Applied Cybersecurity ATHENE, Germany(图宾根大学计算机科学系通用知识处理实验室及应用网络安全国家研究中心ATHENE德国)
PaperAudit-Bench: Benchmarking Error Detection in Research Papers for Critical Automated Peer Review
Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所);Pengcheng Laboratory(鹏城实验室);University of Chinese Academy of Sciences(中国科学院大学);Wenge Technology(文英科技)
WarrantScore: Modeling Warrants between Claims and Evidence for Substantiation Evaluation in Peer Reviews
Nara Institute of Science and Technology(奈良科学技术研究所);OMRON SINIC X Corporation(OMRON SINIC X公司);Institute of Science Tokyo(东京科学研究所);Guardian Robot Project, RIKEN(RIKEN守护机器人项目)
When Reviews Disagree: Fine-Grained Contradiction Analysis in Scientific Peer Reviews
Department of Computer Science and Engineering, Indian Institute of Technology Patna, India(印度理工学院帕纳瓦分校计算机科学与工程系);School of Computer Engineering, KIIT Deemed to be University, Bhubaneswar, India(比哈尔邦布尔萨大学计算机工程学院)
FMMD: A multimodal multidisciplinary dataset of open peer reviews from F1000Research
School of Computer Science and Engineering, Guangzhou Institute of Science and Technology(计算机科学与工程学院,广州科学与技术研究所);College of Foreign Languages and Cultures, Xiamen University(外语学院,厦门大学);Science and Education Evaluation Lab, Guangzhou Institute of Science and Technology(科学与教育评估实验室,广州科学与技术研究所)