The rapid development of science and technology has been accompanied by an exponential growth in peer-reviewed scientific publications. At the same time, the review of each paper is a laborious process that must be carried out by subject matter experts. Thus, providing high-quality reviews of this growing number of papers is a significant challenge. In this work, we ask the question "can we automate scientific reviewing?", discussing the possibility of using state-of-the-art natural language processing (NLP) models to generate first-pass peer reviews for scientific papers. Arguably the most difficult part of this is defining what a "good" review is in the first place, so we first discuss possible evaluation measures for such reviews. We then collect a dataset of papers in the machine learning domain, annotate them with different aspects of content covered in each review, and train targeted summarization models that take in papers to generate reviews. Comprehensive experimental results show that system-generated reviews tend to touch upon more aspects of the paper than human-written reviews, but the generated text can suffer from lower constructiveness for all aspects except the explanation of the core ideas of the papers, which are largely factually correct. We finally summarize eight challenges in the pursuit of a good review generation system together with potential solutions, which, hopefully, will inspire more future research on this subject. We make all code, and the dataset publicly available: https://github.com/neulab/ReviewAdvisor, as well as a ReviewAdvisor system: http://review.nlpedia.ai/.
LLM-Based Scientific Peer Review: Methods, Benchmarks, and Reliability Challenges
L3S Research Center, Leibniz University Hannover(莱布尼茨汉诺威大学L3S研究中心);Peter L. Reichertz Institute for Medical Informatics of TU Braunschweig and Hannover Medical School(布伦瑞克工业大学与汉诺威医学院彼得·L·赖歇茨医学信息学研究所);Lower Saxony Center for AI and Causal Methods in Medicine (CAIMed)(下萨克森州医学人工智能与因果方法中心(CAIMed))
ReviewRL: Towards Automated Scientific Review with RL
Tsinghua University(清华大学);University of Washington(华盛顿大学);Shanghai AI Laboratory(上海人工智能实验室);Peking University(北京大学);Harbin Engineering University(哈尔滨工程大学);Beijing Institute of Technology(北京理工大学)
Are Non-English Papers Reviewed Fairly? Language-of-Study Bias in NLP Peer Reviews
Koç University(科奇大学);KUIS AI Lab(KUIS人工智能实验室);LMU Munich(慕尼黑大学);Munich Center for Machine Learning(慕尼黑机器学习中心);UCLouvain(鲁汶大学);Friedrich-Alexander-Universität Erlangen-Nürnberg(埃尔朗根-纽伦堡大学)
ProReviewer: Evidence-Driven Investigation for Scientific Peer Review
Ubiquitous Knowledge Processing Lab (UKP Lab), Technical University of Darmstadt(达姆施塔特工业大学通用知识处理实验室);National Research Center for Applied Cybersecurity ATHENE, Germany(德国国家应用网络安全研究中心 ATHENE);School of Computer Science, University of Birmingham(伯明翰大学计算机科学学院)
PaperAudit-Bench: Benchmarking Error Detection in Research Papers for Critical Automated Peer Review
Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所);Pengcheng Laboratory(鹏城实验室);University of Chinese Academy of Sciences(中国科学院大学);Wenge Technology(文英科技)
What Can Natural Language Processing Do for Peer Review?
Technical University of Darmstadt(达姆施塔特工业大学);Mohamed bin Zayed University of Artificial Intelligence(穆罕默德·本·扎耶德人工智能大学);Radboud University Nijmegen(奈梅亨拉德堡德大学);Carnegie Mellon University(卡内基梅隆大学);Allen Institute for Artificial Intelligence(艾伦人工智能研究所);Bocconi University(博科尼大学);The University of Sydney(悉尼大学);Universität Hamburg(汉堡大学);University of British Columbia(不列颠哥伦比亚大学);University of Applied Sciences Darmstadt(达姆施塔特应用科学大学);Université Paris-Saclay(巴黎萨克雷大学);Monash University(莫纳什大学);The Hebrew University of Jerusalem(耶路撒冷希伯来大学);University of Washington(华盛顿大学);Georgia Institute of Technology(佐治亚理工学院);IT University of Copenhagen(哥本哈根信息技术大学)
Reviewing the Reviewer: LLM-Assisted Reviewer Feedback Generation for Guideline Compliance
Ubiquitous Knowledge Processing Lab (UKP Lab), Department of Computer Science, Technical University of Darmstadt and National Research Center for Applied Cybersecurity ATHENE, Germany(技术大学达姆施塔特计算机科学系 Ubiquitous Knowledge Processing Lab(UKP Lab)及国家应用网络安全研究中心 ATHENE,德国);Department of Data Science & AI, Monash University, Australia(墨尔本大学数据科学与人工智能系,澳大利亚);Data Science Group, University of Hamburg(汉堡大学数据科学组)
CodeScientist: End-to-End Semi-Automated Scientific Discovery with Code-based Experimentation
Allen Institute for Artificial Intelligence(艾伦人工智能研究院);University of Washington(华盛顿大学);University of Arizona(亚利桑那大学);Hebrew University of Jerusalem(耶路撒冷希伯来大学)
Using Large Language Models to Automate Category and Trend Analysis of Scientific Articles: An Application in Ophthalmology
Gazi University(加齐大学);Aristotle University of Thessaloniki (AUTH)(亚里士多德大学);Association to Prevent Blindness in Mexico(墨西哥预防失明协会);South Valley University(南谷大学);University of Maryland, School of Medicine(马里兰大学医学院)
SciToolAgent: A Knowledge Graph-Driven Scientific Agent for Multi-Tool Integration
College of Computer Science and Technology(计算机科学与技术学院);Zhejiang University(浙江大学);Zhejiang Key Laboratory of Intelligent Manufacturing for Functional Chemicals(功能化学品智能制造重点实验室);ZJU-Hangzhou Global Scientific and Technological Innovation Center(浙大杭州国际科学技术创新中心);ZJU-UIUC Institute(浙大UIUC研究院);The Polytechnic Institute(技术学院);State Key Laboratory of Ocean Sensing(海洋感知国家重点实验室)