arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2109.01190cs.CL
收藏

从偏好学习的角度看学术同行评议中的辅助决策

Assisting Decision Making in Scholarly Peer Review: A Preference Learning Perspective

University of Bristol(布里斯托大学);Technische Universität Darmstadt(达姆施塔特工业大学)

Nils Dycke, Edwin Simpson, Ilia Kuznetsov, Iryna Gurevych

摘要:

同行评审是学术界质量控制的主要手段;作为同行评审过程的结果,项目和区域主席根据他们收到的评审报告和分数对每篇论文做出接受决定。科学工作的质量是多方面的;再加上审查的主观性,这使得最终决策变得困难和耗时。为了支持同行评审的最后一步,我们将其形式化为论文排名问题。我们引入了一种新颖的,多方面的通用评估框架排名提交同行评议的基础上,考虑到有效性,效率和公平性。我们提出了一个偏好学习的任务,同时考虑审查文本和分数,以减轻不可避免的偏见和噪音的评论。我们对ACL 2018会议的同行评审数据进行的实验表明,我们基于偏好学习的方法优于基线和先前的工作,同时强调了使用评审文本和分数对提交内容进行排名的重要性。

英文摘要:

Peer review is the primary means of quality control in academia; as an outcome of a peer review process, program and area chairs make acceptance decisions for each paper based on the review reports and scores they received. Quality of scientific work is multi-faceted; coupled with the subjectivity of reviewing, this makes final decision making difficult and time-consuming. To support this final step of peer review, we formalize it as a paper ranking problem. We introduce a novel, multi-faceted generic evaluation framework for ranking submissions based on peer reviews that takes into account effectiveness, efficiency and fairness. We propose a preference learning perspective on the task that considers both review texts and scores to alleviate the inevitable bias and noise in reviews. Our experiments on peer review data from the ACL 2018 conference demonstrate the superiority of our preference-learning-based approach over baselines and prior work, while highlighting the importance of using both review texts and scores to rank submissions.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑