大型会议中合格审稿人稀缺问题的新审稿人实验
A Novice-Reviewer Experiment to Address Scarcity of Qualified Reviewers in Large Conferences
摘要:
会议同行评审是一个人工计算的过程,其重要性怎么强调都不过分:它不仅确定了最好的提交材料,而且最终通过促进一些想法和限制其他想法来影响整个研究领域的未来。主要人工智能会议收到的提交材料数量激增,增加了合格评审人员的负担,从而对评审过程的可持续性提出了挑战,而合格评审人员的增长速度要慢得多。在这项工作中,我们考虑的问题,审查招聘的重点是在大型会议合格的审查员的稀缺性。具体而言,我们设计了一个程序,用于(i)从主要会议通常不涵盖的人群中招募审查人员,以及(ii)指导他们通过审查管道。结合ICML 2020 -一个大型的顶级机器学习会议-我们通过我们的程序招募了一小部分评审员,并将他们的表现与ICML评审员的一般人群进行比较。我们的实验表明,相结合的招聘和指导机制,允许一个原则性的增强的审查池和结果的审查质量优越的传统的审查相比,由高级成员的程序委员会(元审查)。
英文摘要:
Conference peer review constitutes a human-computation process whose importance cannot be overstated: not only it identifies the best submissions for acceptance, but, ultimately, it impacts the future of the whole research area by promoting some ideas and restraining others. A surge in the number of submissions received by leading AI conferences has challenged the sustainability of the review process by increasing the burden on the pool of qualified reviewers which is growing at a much slower rate. In this work, we consider the problem of reviewer recruiting with a focus on the scarcity of qualified reviewers in large conferences. Specifically, we design a procedure for (i) recruiting reviewers from the population not typically covered by major conferences and (ii) guiding them through the reviewing pipeline. In conjunction with ICML 2020 -- a large, top-tier machine learning conference -- we recruit a small set of reviewers through our procedure and compare their performance with the general population of ICML reviewers. Our experiment reveals that a combination of the recruiting and guiding mechanisms allows for a principled enhancement of the reviewer pool and results in reviews of superior quality compared to the conventional pool of reviews as evaluated by senior members of the program committee (meta-reviewers).
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。