Peer review is the primary means of quality control in academia; as an outcome of a peer review process, program and area chairs make acceptance decisions for each paper based on the review reports and scores they received. Quality of scientific work is multi-faceted; coupled with the subjectivity of reviewing, this makes final decision making difficult and time-consuming. To support this final step of peer review, we formalize it as a paper ranking problem. We introduce a novel, multi-faceted generic evaluation framework for ranking submissions based on peer reviews that takes into account effectiveness, efficiency and fairness. We propose a preference learning perspective on the task that considers both review texts and scores to alleviate the inevitable bias and noise in reviews. Our experiments on peer review data from the ACL 2018 conference demonstrate the superiority of our preference-learning-based approach over baselines and prior work, while highlighting the importance of using both review texts and scores to rank submissions.
Large Language Models Penetration in Scholarly Writing and Peer Review
The Chinese University of Hong Kong, Shenzhen(香港中文大学(深圳));University of Electronic Science and Technology of China(电子科技大学);Wuhan University of Technology(武汉理工大学);University of Copenhagen(哥本哈根大学);Shenzhen Research Institute of Big Data(深圳大数据研究院)
Beyond "Not Novel Enough": Enriching Scholarly Critique with LLM-Assisted Feedback
UKP Lab, TU Darmstadt and Hessian Center for AI (hessian.AI)(图林人工智能实验室、德累斯顿理工大学和黑森人工智能中心);MBZUAI(马克斯·普朗克人工智能研究所);The Allen Institute for AI (AI2)(人工智能研究院)
Are We There Yet? Revealing the Risks of Utilizing Large Language Models in Scholarly Peer Review
Shanghai Jiao Tong University(上海交通大学);Georgia Institute of Technology(佐治亚理工学院);Shanghai AI Laboratory(上海人工智能实验室);University of Georgia(佐治亚大学);Oxford University(牛津大学)
Automated scholarly paper review: Concepts, technologies, and challenges
Xiamen University(厦门大学);Key Laboratory of Digital Protection and Intelligent Processing of Intangible Cultural Heritage of Fujian and Taiwan, Ministry of Culture and Tourism(文化和旅游部闽台非物质文化遗产数字化保护与智能处理重点实验室)
GLIMPSE: Pragmatically Informative Multi-Document Summarization for Scholarly Reviews
International Laboratory on Learning Systems(国际学习系统实验室);MILA - Quebec AI Institute(MILA - 魁北克人工智能研究所);McGill University(麦吉尔大学);Université Paris-Saclay(巴黎萨克雷大学);CNRS, CentraleSupélec(法国国家科学研究中心、中央高等电力学院)
NLPeer: A Unified Resource for the Computational Study of Peer Review
Ubiquitous Knowledge Processing Lab (UKP Lab)(普适知识处理实验室(UKP实验室));Department of Computer Science(计算机科学系);Hessian Center for AI (hessian.AI)(黑森人工智能中心(hessian.AI));Technical University of Darmstadt(达姆施塔特工业大学)
CoCoNUTS: Concentrating on Content while Neglecting Uninformative Textual Styles for AI-Generated Peer Review Detection
Chinese Information Processing Laboratory, Institute of Software, Chinese Academy of Sciences(中国科学院软件研究所中文信息处理实验室);University of Chinese Academy of Sciences(中国科学院大学)
Extending AI for Research to the Humanities: A Multi-Agent Framework for Evidence-Grounded Scholarship
Department of Information Management(信息管理系);Research Center for Digital Humanities(数字人文研究中心);School of Foreign Languages(外国语言学院);Institute for Artificial Intelligence(人工智能研究院)
TreeReview: A Dynamic Tree of Questions Framework for Deep and Efficient LLM-based Scientific Peer Review
National Science Library, Chinese Academy of Sciences(中国科学院国家科学图书馆);Department of Information Resources Management, School of Economics and Management, University of Chinese Academy of Sciences(中国科学院大学经济与管理学院信息资源管理系);The University of Hong Kong(香港大学);Tsinghua University(清华大学);Hong Kong University of Science and Technology (Guangzhou)(香港科学与技术大学(广州))
Structure-Tags Improve Text Classification for Scholarly Document Quality Prediction
Center for Information Technology, University of Groningen(格罗宁根大学信息技术中心);Kapteyn Astronomical Institute, University of Groningen(格罗宁根大学卡普泰恩天文研究所)
在长文本(特别是学术文档)上训练递归神经网络会给学习带来问题。虽然层次注意力网络(HAN)在解决这些问题方面是有效的,但它们仍然丢失了关于文本结构的重要信息。为了解决这些问题,我们建议使用HAN与结构标签相结合,标记文档中句子的作用。为句子添加标签,将它们标记为与标题,摘要或主体文本相对应,从而改进了学术文档质量预测的最新技术。所提出的系统被应用到接受/拒绝的PeerRead数据集上的预测任务,并与最近的基于BiLSTM的模型和联合文本+视觉模型以及普通HAN进行比较。与普通HAN相比,在所有三个域上的准确性都有所提高。在计算和语言领域,我们的新模型总体上效果最好,比最佳文献结果提高了4.7%的准确性。我们还在引入用于预测我们从Allen AI S2 ORC数据集编译的88 k科学出版物的引用数量的标签时获得了改进。对于我们的HAN系统与结构标签,我们达到了28.5%的解释方差,比我们的基于BiLSTM的模型的重新实现提高了1.8%,比普通HAN提高了1.0%。
MOPRD: A multidisciplinary open peer review dataset
School of Informatics, Xiamen University(厦门大学信息学院);Key Laboratory of Digital Protection and Intelligent Processing of Intangible Cultural Heritage of Fujian and Taiwan, Ministry of Culture and Tourism(文化和旅游部闽台非物质文化遗产数字化保护与智能处理重点实验室);College of Foreign Languages and Cultures, Xiamen University(厦门大学外文学院)
Large language models for automated scholarly paper review: A survey
School of Computer Science and Engineering, Guangzhou Institute of Science and Technology(计算机科学与工程学院,科学与技术研究院);School of Economics and Management, Guizhou Normal University(经济学与管理学院,贵州师范大学);School of Artificial Intelligence, Guangzhou Institute of Science and Technology(人工智能学院,科学与技术研究院)
FMMD: A multimodal multidisciplinary dataset of open peer reviews from F1000Research
School of Computer Science and Engineering, Guangzhou Institute of Science and Technology(计算机科学与工程学院,广州科学与技术研究所);College of Foreign Languages and Cultures, Xiamen University(外语学院,厦门大学);Science and Education Evaluation Lab, Guangzhou Institute of Science and Technology(科学与教育评估实验室,广州科学与技术研究所)
'Quis custodiet ipsos custodes?' Who will watch the watchmen? On Detecting AI-generated peer-reviews
Indian Institute of Technology Patna(印度理工学院巴特那分校);National Center for Computational Sciences, Oak Ridge National Laboratory(橡树岭国家实验室国家计算科学中心);School of AI and Data Science, IIT Jodhpur(焦特布尔印度理工学院人工智能与数据科学学院)