同行评审的同行评审：一项随机对照试验及其他实验
Peer Reviews of Peer Reviews: A Randomized Controlled Trial and Other Experiments
AI总结：
本研究通过NeurIPS 2022的大规模随机对照试验和观测数据分析，发现评审意见的长度会显著影响其质量评分，且作者对推荐自己论文的评审存在正向偏见，评审评估中存在与论文评审相当的不一致性和校准误差。
摘要：
是否有可能可靠地评估同行评审的质量？我们研究这一问题，主要出于两个动机——利用评审质量的评估来激励高质量的评审工作，以及衡量实验中评审质量的变化。我们在机器学习领域的顶级会议NeurIPS 2022上开展了一项大规模研究，邀请（元）评审员和作者对提交论文所收到的评审意见进行评估。首先，我们进行了一项随机对照试验（RCT），以检验评审长度带来的偏差。我们通过添加大量无信息内容来生成加长版的评审意见。对照组参与者评估原始评审意见，而实验组参与者评估人为加长后的版本。我们发现，加长后的评审意见在统计上显著地获得比原始评审意见更高的质量评分。在对观测数据的分析中，我们发现作者对自己论文的推荐接收的评审意见存在正向偏见，即使在控制了评审长度、质量以及每位作者论文数量差异等混杂因素后，这种偏见依然存在。我们还测量了对同一评审意见进行多次评估时的不一致率，范围为28%至32%，这与NeurIPS论文评审员的不一致率相当。此外，我们使用质量评分的线性模型评估了评审评估者的校准误差程度，发现其与NeurIPS论文评审员的校准误差估计值相似。最后，我们估计了在如何将各个标准映射到评审质量总体评分方面的主观意见变异性，发现其与论文评审中的变异性大致相同。我们的结果表明，论文评审中存在的各种问题——不一致性、对无关因素的偏见、校准误差、主观性——同样出现在对评审意见的评审中。
英文摘要：
Is it possible to reliably evaluate the quality of peer reviews? We study this question driven by two primary motivations -- incentivizing high-quality reviewing using assessed quality of reviews and measuring changes to review quality in experiments. We conduct a large scale study at the NeurIPS 2022 conference, a top-tier conference in machine learning, in which we invited (meta)-reviewers and authors to evaluate reviews given to submitted papers. First, we conduct a RCT to examine bias due to the length of reviews. We generate elongated versions of reviews by adding substantial amounts of non-informative content. Participants in the control group evaluate the original reviews, whereas participants in the experimental group evaluate the artificially lengthened versions. We find that lengthened reviews are scored (statistically significantly) higher quality than the original reviews. In analysis of observational data we find that authors are positively biased towards reviews recommending acceptance of their own papers, even after controlling for confounders of review length, quality, and different numbers of papers per author. We also measure disagreement rates between multiple evaluations of the same review of 28%-32%, which is comparable to that of paper reviewers at NeurIPS. Further, we assess the amount of miscalibration of evaluators of reviews using a linear model of quality scores and find that it is similar to estimates of miscalibration of paper reviewers at NeurIPS. Finally, we estimate the amount of variability in subjective opinions around how to map individual criteria to overall scores of review quality and find that it is roughly the same as that in the review of papers. Our results suggest that the various problems that exist in reviews of papers -- inconsistency, bias towards irrelevant factors, miscalibration, subjectivity -- also arise in reviewing of reviews.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
