arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 1804.09635cs.CLcs.AI
收藏

同行评议数据集(PeerRead):收集、见解和NLP应用

A Dataset of Peer Reviews (PeerRead): Collection, Insights and NLP Applications

Carnegie Mellon University(卡内基梅隆大学);Allen Institute for Artificial Intelligence(艾伦人工智能研究所);University of Washington(华盛顿大学)

Dongyeop Kang, Waleed Ammar, Bhavana Dalvi, Madeleine van Zuylen, Sebastian Kohlmeier, Eduard Hovy, Roy Schwartz

摘要:

同行评审是科学出版过程中的一个核心组成部分。我们提出了第一个可用于研究目的的科学同行评议公共数据集(PeerRead v1),为研究这一重要工件提供了机会。该数据集包括14.7K纸质草稿以及顶级场馆(包括ACL、NIPS和ICLR)的相应接受/拒绝决策。该数据集还包括专家为论文子集撰写的10.7K文本同行评审。我们描述了数据汇集过程,并报告了同行评审中观察到的有趣现象。我们还基于该数据集提出了两个新的NLP任务,并提供了简单的基线模型。在第一个任务中,我们展示了简单的模型可以预测一篇论文是否被接受,与大多数基线相比,错误减少了21%。在第二个任务中,我们预测的数值分数的审查方面,并表明,简单的模型可以优于平均基线方面的高方差,如“原创性”和“影响力”。

英文摘要:

Peer reviewing is a central component in the scientific publishing process. We present the first public dataset of scientific peer reviews available for research purposes (PeerRead v1) providing an opportunity to study this important artifact. The dataset consists of 14.7K paper drafts and the corresponding accept/reject decisions in top-tier venues including ACL, NIPS and ICLR. The dataset also includes 10.7K textual peer reviews written by experts for a subset of the papers. We describe the data collection process and report interesting observed phenomena in the peer reviews. We also propose two novel NLP tasks based on this dataset and provide simple baseline models. In the first task, we show that simple models can predict whether a paper is accepted with up to 21% error reduction compared to the majority baseline. In the second task, we predict the numerical scores of review aspects and show that simple models can outperform the mean baseline for aspects with high variance such as 'originality' and 'impact'.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑