arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2510.02027cs.AIcs.ET
收藏

模拟学术同行评审的零样本推理

Zero-shot reasoning for simulating scholarly peer-review

College of Engineering and Technology(工程与技术学院);Arab Academy for Science, Technology, and Maritime Transport(阿拉伯科学、技术与海运学院)

Khalid M. Saqr

AI总结:

本研究构建同行评审模拟基准 xPeer,对比其与人类评审的特征差异,发现 xPeer 报告针对性等更优,人类报告显式推理等更强,成果存档于 Zenodo。

摘要:

学术出版需要可扩展的审查,并辅以可审计的证据。本文提出了由两部分组成的基准 xPeer,即通过 this http URL 网页前端提供的同行评审模拟引擎。操作组件分析了 500 条符合稳定任务标准、跨学科和评审模式的模拟记录中的 352 条。人类参考组件发布了 1108 份 F1000Research 手稿的第 1 版记录,附带关联的人类报告,并应用预先指定的“两人类/两 xPeer”比较。人类评审的文本和建议未纳入生成输入,源连接在 xPeer 输出持久化后进行,此流程定义了工作流级别的评审 withhold( withholding 译为“ withhold 操作”?不,按规则,此处 workflow-level review withholding 译为“工作流级评审 withhold”?不,规则里的术语:abstain 是弃权(不执行),其他术语保留原名。哦,原规则:算法、模型、框架、数据集、基准和项目的正式名称必须保留英文原名,缩写保留原大小写。数字、百分比等准确保留。那重新翻译:学术出版需要可扩展的审查,并辅以可审计的证据。本文提出了由两部分组成的基准 xPeer,即通过 this http URL 网页前端提供的同行评审模拟引擎。操作组件分析了 500 条符合稳定任务标准、跨学科和评审模式的模拟记录中的 352 条。人类参考组件发布了 1108 份 F1000Research 手稿的第 1 版记录,附带关联的人类报告,并应用预先指定的两人类/两 xPeer 比较。人类评审的文本和建议未纳入生成输入,源连接在 xPeer 输出持久化后进行,此流程定义了工作流级评审 withholding;先前的模型暴露不在记录的设计范围内。在恰好有两份人类报告的 802 条记录中,271 条包含两条可用的 xPeer 评审员字段,完整对可用性率为 33.8%。在确定性提取规则下,手稿级报告的中位数长度:xPeer 为 1889 词,人类为 763 词;中位数关注点数量分别为 41 和 13。xPeer 报告表现出更高的针对性、类别覆盖率和可执行性;人类报告则表现出更高的显式推理语言、词汇手稿证明、基于分类学的科学相关性,以及更低的源内平均冗余度。跨源词汇关注点匹配和建议一致性较低。因此,该证据定义了不同的可观测评审特征和透明的可复现性基线。单个关注点的科学正确性、自主编辑使用及跨系统优势需要专家裁决和通用协议测试。该研究级数据集和精确版本固定的可复现性记录存档于 Zenodo。

英文摘要:

Scholarly publishing requires scalable scrutiny supported by auditable evidence. This paper presents a two-component benchmark of xPeer, the peer-review simulation engine delivered through the xPeerd.com web front-end. The operational component analyzes 352 of 500 simulation records retained under stable-task criteria across disciplines and review modes. The human-reference component releases 1,108 version-1 F1000Research manuscript records with linked human reports and applies a prespecified two-human/two-xPeer comparison. Human-review text and recommendations remained outside the generation input, and source joining occurred after xPeer outputs had been persisted. This procedure defines workflow-level review withholding; prior model exposure falls outside the recorded design. Among 802 records with exactly two human reports, 271 contained two usable xPeer reviewer fields, giving a complete-pair availability rate of 33.8%. Under deterministic extraction rules, median manuscript-level report length was 1,889 words for xPeer and 763 for humans, while median concern count was 41 and 13, respectively. xPeer reports showed higher targeting, category coverage, and executability. Human reports showed higher explicit-reasoning language, lexical manuscript attestation, taxonomy-based scientific relevance, and lower mean within-source redundancy. Cross-source lexical concern matching and recommendation agreement were low. The evidence therefore defines distinct observable review profiles and a transparent reproducibility baseline. Scientific correctness of individual concerns, autonomous editorial use, and cross-system superiority require expert adjudication and common-protocol testing. The study-level dataset and exact version-pinned reproducibility record are archived on Zenodo

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑