同行评议中的不一致性：2014年NeurIPS实验的回顾
Inconsistency in Conference Peer Review: Revisiting the 2014 NeurIPS Experiment
摘要：
在本文中，我们回顾了2014年的NeurIPS实验，该实验研究了会议同行评审中的不一致性。我们确定50%的评审员质量评分的变化是主观的。此外，自实验以来已经过去了七年，我们发现对于\emph{accepted}论文，质量分数和论文的影响力之间没有相关性，作为引用计数的函数。我们追踪被拒绝的论文的命运，恢复这些论文最终发表的地方。对于这些论文，我们发现质量分数和影响力之间存在相关性。我们的结论是，2014年会议的审查过程是好的识别差的论文，但穷人识别好的论文。我们提出了一些改进审查过程的建议，但也警告不要删除主观因素。最后，我们建议，实验的真正结论是，社会应该把较少的责任在“顶级会议出版物”的概念时，评估个人研究人员的质量。对于NeurIPS 2021，PC正在重复实验，并进行新的实验。
英文摘要：
In this paper we revisit the 2014 NeurIPS experiment that examined inconsistency in conference peer review. We determine that 50\% of the variation in reviewer quality scores was subjective in origin. Further, with seven years passing since the experiment we find that for \emph{accepted} papers, there is no correlation between quality scores and impact of the paper as measured as a function of citation count. We trace the fate of rejected papers, recovering where these papers were eventually published. For these papers we find a correlation between quality scores and impact. We conclude that the reviewing process for the 2014 conference was good for identifying poor papers, but poor for identifying good papers. We give some suggestions for improving the reviewing process but also warn against removing the subjective element. Finally, we suggest that the real conclusion of the experiment is that the community should place less onus on the notion of `top-tier conference publications' when assessing the quality of individual researchers. For NeurIPS 2021, the PCs are repeating the experiment, as well as conducting new ones.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
