arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 1808.09057cs.AIcs.GTcs.LG
收藏

损失函数、公理和同行评审

Loss Functions, Axioms, and Peer Review

Carnegie Mellon University(卡内基梅隆大学);Harvard University(哈佛大学)

Ritesh Noothigattu, Nihar B. Shah, Ariel D. Procaccia

摘要:

通常会看到少数审稿人拒绝一篇高度新颖的论文,因为他们认为,比如说,广泛的实验比新颖性更重要,而整个社区会接受这篇论文。更一般地说,不同的评审员将标准分数映射到最终建议是同行评审不一致的主要原因。在本文中,我们提出了一个框架的启发经验风险最小化(ERM)学习社区的聚合映射。出现的关键挑战是为机构风险管理规定一个损失函数。我们考虑一类$L(p,q)$损失函数,这是一个矩阵扩展的标准类$L_p$损失的向量;在这里的损失函数的选择相当于选择超参数$p, q \in [1,\infty]$。为了解决我们的问题中缺乏基础事实的问题,我们转而利用计算社会选择来确定超参数$p$和$q$的理想值。具体来说,我们表征$p=q=1$作为这些超参数,满足三个自然公理性质的唯一选择。最后,我们实施并应用我们的方法来审查IJCAI 2017。

英文摘要:

It is common to see a handful of reviewers reject a highly novel paper, because they view, say, extensive experiments as far more important than novelty, whereas the community as a whole would have embraced the paper. More generally, the disparate mapping of criteria scores to final recommendations by different reviewers is a major source of inconsistency in peer review. In this paper we present a framework inspired by empirical risk minimization (ERM) for learning the community's aggregate mapping. The key challenge that arises is the specification of a loss function for ERM. We consider the class of $L(p,q)$ loss functions, which is a matrix-extension of the standard class of $L_p$ losses on vectors; here the choice of the loss function amounts to choosing the hyperparameters $p, q \in [1,\infty]$. To deal with the absence of ground truth in our problem, we instead draw on computational social choice to identify desirable values of the hyperparameters $p$ and $q$. Specifically, we characterize $p=q=1$ as the only choice of these hyperparameters that satisfies three natural axiomatic properties. Finally, we implement and apply our approach to reviews from IJCAI 2017.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑