打破静态评估下评审可靠性的幻觉：面向基于LLM的科学评审者的SCOPE模糊测试

中文摘要 AI 辅助

提交论文数量的快速增长和评审工作量的增加加速了大型语言模型（LLMs）在同行评审中的应用。先前的研究表明，基于LLM的评审者能够惩罚内容扰动，例如过度声称，这表明其具有一定程度的可靠性。然而，这些结论主要基于使用静态模板实例化的狭窄扰动策略集，提供的实际可靠性证据有限。在本文中，我们构建了一个三层评估框架，涵盖表面呈现、论证逻辑和价值判断的扰动。对代表性基于LLM的评审者的实验揭示了静态评估的两个局限性：分层脆弱性，即扰动效果取决于论文原始评审分数的高低；以及扰动覆盖不足，即单一模板无法覆盖多样化实现所暴露的脆弱性。为解决这些局限性，我们提出了SCOPE-Fuzzer，一种策略感知的模糊测试器，它结合了反馈驱动的策略选择与论文内容的自适应变异。通过迭代地使用动态扰动探测评审者，SCOPE-Fuzzer持续发现静态评估和其他基线方法所忽视的脆弱性。

英文摘要

The rapid growth of submissions and reviewing workload has accelerated the use of large language models (LLMs) in peer review. Prior studies suggest that LLM-based reviewers can penalize content perturbations, such as overclaiming, indicating a certain degree of reliability. Yet these conclusions are largely based on a narrow set of perturbation strategies instantiated with static templates, providing limited evidence of actual reliability. In this paper, we construct a three-level evaluation framework covering perturbations to surface presentation, argumentative logic, and value judgment. Experiments on representative LLM-based reviewers reveal two limitations of static evaluation: stratified vulnerability, where perturbation effects depend on whether the paper's original review score is high or low, and perturbation undercoverage, where a single template misses vulnerabilities exposed by diverse realizations. To address these limitations, we propose SCOPE-Fuzzer, a strategy-aware fuzzer that combines feedback-driven strategy selection with adaptive mutation of paper content. By iteratively probing reviewers with dynamic perturbations, SCOPE-Fuzzer consistently uncovers vulnerabilities overlooked by static evaluation and other baselines.

发表机构

• Wuhan University（武汉大学）
• East China Normal University（华东师范大学）

机构由 AI 辅助整理，请以论文原文为准。