基于开放同行评审中个体智慧度量的论文质量评估
Paper Quality Assessment based on Individual Wisdom Metrics from Open Peer Review
AI总结：
针对传统封闭同行评审的诸多弊端，本研究基于会议数据提出开放自下而上的评审模式，通过贝叶斯方法和评审者互评机制提升论文质量评估效果，并设计激励机制以推动评审流程更高效公平透明。
摘要：
传统封闭式同行评审体系在科学出版中一直发挥着核心作用，但它往往存在速度慢、成本高、不透明、随机性强且可能存在偏见等问题——这些因素会阻碍科学进步，削弱公众信任。在此，我们提出并检验了一种替代型科学同行评审形式的有效性与准确性：通过开放、自下而上的流程实现。首先，利用两大科学会议（CCN2023和ICLR2023）的数据，我们指出评审分数的高变异性和评审者之间的低相关性给集体评审带来了挑战。我们量化了评审者评分与社区共识分数的一致性，并将其作为评审者质量估计指标，结果令人意外地显示，评审者质量分数与作者质量并不相关。相反，我们揭示了一种倒U型关系：论文得分处于中等水平的作者是最佳评审者。我们评估了基于不同个体评审者可靠性评估来估计论文质量的经验贝叶斯方法。研究表明，在一次性“评审后打分”场景中，无论是在我们的模型中还是在真实同行评审数据上，贝叶斯度量相较于简单平均法都显著提升了论文质量评估效果。随后，我们考虑了一个包含出版、评审和打分的持续运行模型，其中评审者不仅要为论文打分，还要为其他评审者打分。我们证明，即便不可靠（但无偏见）的评审者占主导，用户生成的评审者评级也能产生稳健且高质量的论文打分结果。最后，我们概述了激励机制，以表彰高质量评审者，并鼓励对投稿论文开展更广泛的评审覆盖。这些发现表明，自主选择的开放同行评审流程具备可扩展性、可靠性和公平性，有望提升同行评审流程的速度、公平性与透明度。
英文摘要：
Traditional closed peer review systems, which have played a central role in scientific publishing, are often slow, costly, non-transparent, stochastic, and possibly subject to biases - factors that can impede scientific progress and undermine public trust. Here, we propose and examine the efficacy and accuracy of an alternative form of scientific peer review: through an open, bottom-up process. First, using data from two major scientific conferences (CCN2023 and ICLR2023), we highlight how high variability of review scores and low correlation across reviewers presents a challenge for collective review. We quantify reviewer agreement with community consensus scores and use this as a reviewer quality estimator, showing that surprisingly, reviewer quality scores are not correlated with authorship quality. Instead, we reveal an inverted U-shape relationship, where authors with intermediate paper scores are the best reviewers. We assess empirical Bayesian methods to estimate paper quality based on different assessments of individual reviewer reliability. We show how under a one-shot review-then-score scenario, both in our models and on real peer review data, a Bayesian measure significantly improves paper quality assessments relative to simple averaging. We then consider an ongoing model of publishing, reviewing, and scoring, with reviewers scoring not only papers but also other reviewers. We show that user-generated reviewer ratings can yield robust and high-quality paper scoring even when unreliable (but unbiased) reviewers dominate. Finally, we outline incentive structures to recognize high-quality reviewers and encourage broader reviewing coverage of submitted papers. These findings suggest that a self-selecting open peer review process is potentially scalable, reliable, and equitable with the possibility of enhancing the speed, fairness, and transparency of the peer review process.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
