Peerify：同行评审声明验证的基准测试

中文摘要 AI 辅助

同行评审在学术出版中发挥着核心作用，然而验证评审者的声明是否得到稿件证据支持，在很大程度上仍是一个手动且耗时的过程。我们提出了Peerify，一个用于稿件依据的同行评审声明验证的流水线。给定一份稿件和一条评审意见，Peerify流水线将评审意见分解为原子声明，检索相关的稿件证据，并确定每条声明是否得到论文的支持。为了支持该流水线的开发和评估，我们构建了一个包含800条声明的基准，这些声明源自从NeurIPS 2024和ICLR 2024收集的真实同行评审交互，其中包括一个300条声明的人工标注子集，用于审计自动监督。我们在Peerify流水线中评估了最先进的语言模型和检索策略，以及蕴含基线。我们的结果证明了以检索为中心的验证和声明分解的重要性，同时突出了模糊性和解释性评审声明所带来的挑战。自动标签在90.3%的审计声明上与人类共识一致（κ = 0.87），而现成的蕴含模型保持在0.24宏F1以下。

英文摘要

Peer review plays a central role in scholarly publishing, yet verifying whether reviewer claims are supported by manuscript evidence remains a largely manual and time-consuming process. We present Peerify, a pipeline for manuscript-grounded verification of peer-review claims. Given a manuscript and a review comment, the Peerify pipeline decomposes reviews into atomic claims, retrieves relevant manuscript evidence, and determines whether each claim is supported by the paper. To support the development and evaluation of the pipeline, we construct a benchmark of 800 claims derived from authentic peer-review interactions collected from NeurIPS 2024 and ICLR 2024, including a 300-claim hand-labeled subset used to audit the automated supervision. We evaluate state-of-the-art language models and retrieval strategies within the Peerify pipeline, together with entailment baselines. Our results demonstrate the importance of retrieval-centered verification and claim decomposition, while highlighting the challenges posed by ambiguous and interpretive reviewer claims. Automated labels agree with human consensus on 90.3% of audited claims ($κ= 0.87$), while off-the-shelf entailment models stay below 0.24 macro-F1.

发表机构

• Reviewerly
• University of Toronto（多伦多大学）
• University of California, Berkeley（加利福尼亚大学伯克利分校）

机构由 AI 辅助整理，请以论文原文为准。