arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2303.09020cs.GT
收藏

会议同行评审的系统层面分析

A System-Level Analysis of Conference Peer Review

Rutgers University(罗格斯大学);George Mason University(乔治梅森大学);University of Michigan(密歇根大学);University of Southern California(南加州大学)

Yichi Zhang, Fang-Yi Yu, Grant Schoenebeck, David Kempe

AI总结:

本文将会议与作者互动建模为 Stackelberg 博弈,分析录用门槛、审稿数量和重投政策,揭示“重新投稿差距”及其对审稿负担与会议质量的影响。

摘要:

会议同行评审过程涉及三类具有不同目标的参与者:作者希望自己的论文被有声望的会议录用(并且越快越好),会议希望呈现一个包含大量高质量论文、少量低质量论文的程序,而审稿人则希望避免因审稿而负担过重。这些目标远未协调一致,主要原因是对投稿的评价本身就带有噪声。多年来,各会议尝试了许多政策来权衡这些取舍。这些尝试包括设定不同的录用门槛、改变每篇投稿的审稿数量、要求在重新投稿时附上此前的审稿意见,以及其他做法。在本研究中,我们从分析和实证两个方面考察各类政策的效果如何,更重要的是,考察它们有效或无效的原因。\n\n我们将会议与作者之间的互动建模为一个 Stackelberg 博弈:在该博弈中,一个有声望的会议承诺采用某种录用政策;作者则在每一轮评审中作出最佳回应,即向该会议(重新)投稿或不(重新)投稿,另一种选择是“必然录用”(例如审稿较宽松的发表场所)。我们的主要结果包括以下观察:1)会议通常应将录用阈值设定得高于其实际期望的质量;我们将此称为“重新投稿差距”。2)审稿负担在很大程度上由临界论文的重新投稿所驱动——因此,明智地选择录用阈值,可能会减少审稿数量,同时只给会议质量带来可接受的损失。3)会议声望、审稿人的不准确性以及作者的耐心都会增大重新投稿差距,从而在会议质量固定的情况下增加审稿负担。为保证稳健性,我们进一步考虑了不同的论文质量模型,并将理论结果与基于真实数据估计的合理参数所进行的仿真进行了比较。

英文摘要:

The conference peer review process involves three constituencies with different objectives: authors want their papers accepted at prestigious venues (and quickly), conferences want to present a program with many high-quality and few low-quality papers, and reviewers want to avoid being overburdened by reviews. These objectives are far from aligned, primarily because the evaluation of a submission is inherently noisy. Over the years, conferences have experimented with numerous policies to navigate the tradeoffs. These experiments include setting various bars for acceptance, varying the number of reviews per submission, requiring prior reviews to be included with resubmissions, and others. In this work, we investigate, both analytically and empirically, how well various policies work, and more importantly, why they do or do not work. We model the conference-author interactions as a Stackelberg game in which a prestigious conference commits to an acceptance policy; the authors best-respond by (re)submitting or not (re)submitting to the conference in each round of review, the alternative being a "sure accept" (such as a lightly refereed venue). Our main results include the following observations: 1) the conference should typically set a higher acceptance threshold than the actual desired quality; we call this the "resubmission gap". 2) the reviewing load is heavily driven by resubmissions of borderline papers - therefore, a judicious choice of acceptance threshold may lead to fewer reviews while incurring an acceptable loss in conference quality. 3) conference prestige, reviewer inaccuracy, and author patience increase the resubmission gap, and thus increase the review load for a fixed level of conference quality. For robustness, we further consider different models of paper quality and compare our theoretical results to simulations based on plausible parameters estimated from real data.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑