Peer reviewing is a central component in the scientific publishing process. We present the first public dataset of scientific peer reviews available for research purposes (PeerRead v1) providing an opportunity to study this important artifact. The dataset consists of 14.7K paper drafts and the corresponding accept/reject decisions in top-tier venues including ACL, NIPS and ICLR. The dataset also includes 10.7K textual peer reviews written by experts for a subset of the papers. We describe the data collection process and report interesting observed phenomena in the peer reviews. We also propose two novel NLP tasks based on this dataset and provide simple baseline models. In the first task, we show that simple models can predict whether a paper is accepted with up to 21% error reduction compared to the majority baseline. In the second task, we predict the numerical scores of review aspects and show that simple models can outperform the mean baseline for aspects with high variance such as 'originality' and 'impact'.
NLPeer: A Unified Resource for the Computational Study of Peer Review
Ubiquitous Knowledge Processing Lab (UKP Lab)(普适知识处理实验室(UKP实验室));Department of Computer Science(计算机科学系);Hessian Center for AI (hessian.AI)(黑森人工智能中心(hessian.AI));Technical University of Darmstadt(达姆施塔特工业大学)
An Audit on the Perspectives and Challenges of Hallucinations in NLP
College of Information Sciences and Technology, Pennsylvania State University(宾夕法尼亚州立大学信息科学与技术学院);School of Interactive Computing, Georgia Institute of Technology(佐治亚理工学院交互计算学院);Department of Computer Science & Engineering, Pennsylvania State University(宾夕法尼亚州立大学计算机科学与工程系);Adobe Research(奥多比研究中心)
COMPARE: A Taxonomy and Dataset of Comparison Discussions in Peer Reviews
Engg. Indian Institute of Technology, Gandhinagar Gujarat, India singh\;Engg. Indian Institute of Technology, Gandhinagar Gujarat, India;Engg. Indian Institute of Technology, Kharagpur West Bengal, India
Hierarchical Bi-Directional Self-Attention Networks for Paper Review Rating Recommendation
Department of Computer Science, University of Illinois at Chicago(伊利诺伊大学芝加哥分校计算机科学系);Beijing Advanced Innovation Center for Big Data and Brain Computing, Beihang University(北京航空航天大学大数据与脑计算北京高等创新中心);School of Cyber Science and Technology, Beihang University(北京航空航天大学网络空间安全学院);Department of Computer Science and Engineering, Lehigh University(里海大学计算机科学与工程系)
Yes-Yes-Yes: Proactive Data Collection for ACL Rolling Review and Beyond
Technical University of Darmstadt(达姆施塔特工业大学);Hessian Center for AI (hessian.AI)(黑森人工智能中心)
向公开文本源的转变使语言处理达到了前所未有的规模,但在公共和公开许可的数据稀缺的领域,服务不足。积极主动地收集文本数据用于研究是解决这一不足的可行战略,但缺乏考虑到数据汇集的许多道德、法律和保密方面的系统方法。我们的工作提出了一个案例研究的积极数据汇集在同行评审-一个具有挑战性和资源不足的自然语言处理域。我们概述了主动数据汇集的伦理和法律要求,并介绍了“是-是-是”,第一个基于捐赠的同行评审数据汇集工作流程,满足这些要求。我们报告了ACL Rolling Review中Yes-Yes-Yes的实施情况,并实证研究了主动数据汇集对数据集大小的影响以及同行评审平台上捐赠行为引起的偏差。
What Can Natural Language Processing Do for Peer Review?
Technical University of Darmstadt(达姆施塔特工业大学);Mohamed bin Zayed University of Artificial Intelligence(穆罕默德·本·扎耶德人工智能大学);Radboud University Nijmegen(奈梅亨拉德堡德大学);Carnegie Mellon University(卡内基梅隆大学);Allen Institute for Artificial Intelligence(艾伦人工智能研究所);Bocconi University(博科尼大学);The University of Sydney(悉尼大学);Universität Hamburg(汉堡大学);University of British Columbia(不列颠哥伦比亚大学);University of Applied Sciences Darmstadt(达姆施塔特应用科学大学);Université Paris-Saclay(巴黎萨克雷大学);Monash University(莫纳什大学);The Hebrew University of Jerusalem(耶路撒冷希伯来大学);University of Washington(华盛顿大学);Georgia Institute of Technology(佐治亚理工学院);IT University of Copenhagen(哥本哈根信息技术大学)
Are Non-English Papers Reviewed Fairly? Language-of-Study Bias in NLP Peer Reviews
Koç University(科奇大学);KUIS AI Lab(KUIS人工智能实验室);LMU Munich(慕尼黑大学);Munich Center for Machine Learning(慕尼黑机器学习中心);UCLouvain(鲁汶大学);Friedrich-Alexander-Universität Erlangen-Nürnberg(埃尔朗根-纽伦堡大学)
MuTSE: A Human-in-the-Loop Multi-use Text Simplification Evaluator
University of Bucharest(布加勒斯特大学);Universitatea Creștină "Dimitrie Cantemir"(迪米特里耶·坎泰米尔基督教大学);Cu Drag si Sport SRL(Cu Drag si Sport 有限责任公司);Softbinator Technologies(Softbinator 科技公司)
LazyReview A Dataset for Uncovering Lazy Thinking in NLP Peer Reviews
Ubiquitous Knowledge Processing Lab, Department of Computer Science and Hessian Center for AI (hessian.AI), Technical University of Darmstadt(德意志联邦共和国达姆施塔特技术大学);School of Computing Technologies, Royal Melbourne Institute of Technology, Australia(澳大利亚皇家墨尔本理工学院);Data Science Group, University of Hamburg(汉堡大学);Department of Data Science & AI, Monash University, Australia(澳大利亚墨尔本大学)
NovBench: Evaluating Large Language Models on Academic Paper Novelty Assessment
School of Economics and Management, Nanjing University of Science and Technology(南京理工大学经济管理学院);School of Electronic Engineering and Computer Science, Queen Mary University of London(伦敦玛丽女王大学电子工程与计算机科学学院);School of Management, Anhui University(安徽大学管理学院)
Evolution of ESG-focused DLT Research: An NLP Analysis of the Literature
Centre for Blockchain Technologies, University College London(区块链技术中心,伦敦大学学院);UK Centre for Blockchain Technologies(英国区块链技术中心);Modul University Vienna(维也纳Modul大学);HSBC Business School, Peking University(汇丰商学院,北京大学)
Towards the Roof of the World: NLP Progress in Tibetan Languages
School of Philosophy and Religious Studies, Minzu University of China(中央民族大学哲学与宗教学学院);School of Information Engineering, Minzu University of China(中央民族大学信息工程学院);Institute of National Security, Minzu University of China(中央民族大学国家安全研究院);National Language Resource Monitoring & Research Center — Minority Languages Branch(国家语言资源监测与研究中心少数民族语言分中心);School of Information Science and Technology, Xizang University(西藏大学信息科学技术学院);The State Key Laboratory of Tibetan Intelligence(藏语智能国家重点实验室)
Beyond "Not Novel Enough": Enriching Scholarly Critique with LLM-Assisted Feedback
UKP Lab, TU Darmstadt and Hessian Center for AI (hessian.AI)(图林人工智能实验室、德累斯顿理工大学和黑森人工智能中心);MBZUAI(马克斯·普朗克人工智能研究所);The Allen Institute for AI (AI2)(人工智能研究院)