arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2102.00176cs.CL
收藏

我们能自动化科学审查吗?

Can We Automate Scientific Reviewing?

Carnegie Mellon University(卡内基梅隆大学)

Weizhe Yuan, Pengfei Liu, Graham Neubig

摘要:

随着科学和技术的迅速发展,经同行评审的科学出版物呈指数级增长。同时,每篇论文的评审是一个费力的过程,必须由主题专家进行。因此,为越来越多的论文提供高质量的评论是一项重大挑战。在这项工作中,我们提出了这样一个问题:“我们能自动化科学审查吗?“,讨论了使用最先进的自然语言处理(NLP)模型为科学论文生成首过同行评审的可能性。可以说,最困难的部分是定义什么是“好”的审查摆在首位,所以我们首先讨论可能的评估措施,这样的审查。然后,我们收集机器学习领域的论文数据集,用每篇评论中涵盖的内容的不同方面对它们进行注释,并训练有针对性的摘要模型,这些模型接受论文以生成评论。综合实验结果表明,系统生成的评论往往涉及更多的方面比人类写的评论,但生成的文本可以遭受较低的建设性的所有方面,除了解释的核心思想的论文,这在很大程度上是事实正确的。最后,我们总结了八个挑战,在追求一个良好的评论生成系统连同潜在的解决方案,这,希望能激发更多的未来研究这个问题。我们公开了所有代码和数据集:https://github.com/neulab/ReviewAdvisor,以及ReviewAdvisor系统:http://review.nlpedia.ai/。

英文摘要:

The rapid development of science and technology has been accompanied by an exponential growth in peer-reviewed scientific publications. At the same time, the review of each paper is a laborious process that must be carried out by subject matter experts. Thus, providing high-quality reviews of this growing number of papers is a significant challenge. In this work, we ask the question "can we automate scientific reviewing?", discussing the possibility of using state-of-the-art natural language processing (NLP) models to generate first-pass peer reviews for scientific papers. Arguably the most difficult part of this is defining what a "good" review is in the first place, so we first discuss possible evaluation measures for such reviews. We then collect a dataset of papers in the machine learning domain, annotate them with different aspects of content covered in each review, and train targeted summarization models that take in papers to generate reviews. Comprehensive experimental results show that system-generated reviews tend to touch upon more aspects of the paper than human-written reviews, but the generated text can suffer from lower constructiveness for all aspects except the explanation of the core ideas of the papers, which are largely factually correct. We finally summarize eight challenges in the pursuit of a good review generation system together with potential solutions, which, hopefully, will inspire more future research on this subject. We make all code, and the dataset publicly available: https://github.com/neulab/ReviewAdvisor, as well as a ReviewAdvisor system: http://review.nlpedia.ai/.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑