arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2601.15249cs.LGcs.AIcs.GTstat.ME
收藏

通过等比机制推荐最佳论文奖项:机器学习/人工智能会议

Recommending Best Paper Awards for ML/AI Conferences via the Isotonic Mechanism

Yale University(耶鲁大学);University of Pennsylvania(宾夕法尼亚大学);University of North Carolina at Chapel Hill(北卡罗来纳大学教堂山分校)

Garrett G. Wen, Buxin Su, Natalie Collina, Zhun Deng, Weijie Su

AI总结:

本文提出通过等比机制提高机器学习会议最佳论文评选质量,通过作者评估调整评分以提升真实质量估计。

摘要:

机器学习和人工智能会议,如NeurIPS和ICML,现在经常接收上万份提交,这给维持同行评审过程的质量和一致性带来了重大挑战。这种挑战在最佳论文奖项上尤为突出,因为这些奖项是同行评审过程的重要组成部分,但近年来其选择已成为备受争议的话题。在本文中,我们介绍了一种作者协助机制,以促进最佳论文奖项的评选。我们的方法采用等比机制,以排名的形式获取作者对其提交的评估,随后利用这些评估来调整原始评审分数,以最优估计提交的真实质量。我们证明,当作者的效用是调整后分数的凸加性函数时,作者有动机如实报告,并通过公开可访问的ICLR 2019至2023年和NeurIPS 2021至2023年的评审数据验证了这一凸性假设。关键的是,在作者只有一个配额的情况下,即只能提名一篇论文时,我们证明即使效用函数仅仅是非递减且加性的,诚实性仍然成立。这一发现代表了对先前工作所需假设的显著放宽。在实际应用中,我们扩展了我们的机制以适应常见的作者共同署名情况。最后,模拟结果表明,我们的机制显著提高了获得奖项的论文质量。

英文摘要:

Machine learning and artificial intelligence conferences such as NeurIPS and ICML now regularly receive tens of thousands of submissions, posing significant challenges to maintaining the quality and consistency of the peer review process. This challenge is particularly acute for best paper awards, which are an important part of the peer review process, yet whose selection has increasingly become a subject of debate in recent years. In this paper, we introduce an author-assisted mechanism to facilitate the selection of best paper awards. Our method employs the Isotonic Mechanism for eliciting authors' assessments of their own submissions in the form of a ranking, which is subsequently utilized to adjust the raw review scores for optimal estimation of the submissions' ground-truth quality. We demonstrate that authors are incentivized to report truthfully when their utility is a convex additive function of the adjusted scores, and we validate this convexity assumption for best paper awards using publicly accessible review data of ICLR from 2019 to 2023 and NeurIPS from 2021 to 2023. Crucially, in the special case where an author has a single quota -- that is, may nominate only one paper -- we prove that truthfulness holds even when the utility function is merely nondecreasing and additive. This finding represents a substantial relaxation of the assumptions required in prior work. For practical implementation, we extend our mechanism to accommodate the common scenario of overlapping authorship. Finally, simulation results demonstrate that our mechanism significantly improves the quality of papers selected for awards.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑