LLMs Assist NLP Researchers: Critique Paper (Meta-)Reviewing
University of Illinois Chicago(伊利诺伊大学芝加哥分校);Salesforce Research(Salesforce研究院);Hong Kong Polytechnic University(香港理工大学);Penn State University(宾夕法尼亚州立大学);Google DeepMind(谷歌DeepMind);University of Southern California(南加州大学);University of California, Davis(加州大学戴维斯分校);University of Basel(巴塞尔大学);Xiaomi AI Lab(小米AI实验室);Scale AI;HKUST(香港科技大学);Georgia Institute of Technology(佐治亚理工学院);University of California, Santa Cruz(加州大学圣克鲁兹分校);Singapore University of Technology and Design(新加坡科技设计大学);Texas A&M University(德克萨斯农工大学);Massey University(梅西大学);University of Liverpool(利物浦大学)
Jiangshu Du, Yibo Wang, Wenting Zhao, Zhongfen Deng, Shuaiqi Liu, Renze Lou, Henry Peng Zou, Pranav Narayanan Venkit, Nan Zhang, Mukund Srinath, Haoran Ranran Z… 展开作者
Jiangshu Du, Yibo Wang, Wenting Zhao, Zhongfen Deng, Shuaiqi Liu, Renze Lou, Henry Peng Zou, Pranav Narayanan Venkit, Nan Zhang, Mukund Srinath, Haoran Ranran Zhang, Vipul Gupta, Yinghui Li, Tao Li, Fei Wang, Qin Liu, Tianlin Liu, Pengzhi Gao, Congying Xia, Chen Xing, Jiayang Cheng, Zhaowei Wang, Ying Su, Raj Sanjay Shah, Ruohao Guo, Jing Gu, Haoran Li, Kangda Wei, Zihao Wang, Lu Cheng, Surangika Ranathunga, Meng Fang, Jie Fu, Fei Liu, Ruihong Huang, Eduardo Blanco, Yixin Cao, Rui Zhang, Philip S. Yu, Wenpeng Yin
This work is motivated by two key trends. On one hand, large language models (LLMs) have shown remarkable versatility in various generative tasks such as writing, drawing, and question answering, significantly reducing the time required for many routine tasks. On the other hand, researchers, whose work is not only time-consuming but also highly expertise-demanding, face increasing challenges as they have to spend more time reading, writing, and reviewing papers. This raises the question: how can LLMs potentially assist researchers in alleviating their heavy workload?
This study focuses on the topic of LLMs assist NLP Researchers, particularly examining the effectiveness of LLM in assisting paper (meta-)reviewing and its recognizability. To address this, we constructed the ReviewCritique dataset, which includes two types of information: (i) NLP papers (initial submissions rather than camera-ready) with both human-written and LLM-generated reviews, and (ii) each review comes with "deficiency" labels and corresponding explanations for individual segments, annotated by experts. Using ReviewCritique, this study explores two threads of research questions: (i) "LLMs as Reviewers", how do reviews generated by LLMs compare with those written by humans in terms of quality and distinguishability? (ii) "LLMs as Metareviewers", how effectively can LLMs identify potential issues, such as Deficient or unprofessional review segments, within individual paper reviews? To our knowledge, this is the first work to provide such a comprehensive analysis.
Beyond "Not Novel Enough": Enriching Scholarly Critique with LLM-Assisted Feedback
UKP Lab, TU Darmstadt and Hessian Center for AI (hessian.AI)(图林人工智能实验室、德累斯顿理工大学和黑森人工智能中心);MBZUAI(马克斯·普朗克人工智能研究所);The Allen Institute for AI (AI2)(人工智能研究院)
Decision-Making with Deliberation: Meta-reviewing as a Document-grounded Dialogue
Ubiquitous Knowledge Processing Lab (UKP Lab), Department of Computer Science, Technical University of Darmstadt and National Research Center for Applied Cybersecurity ATHENE, Germany(技术大学达姆施塔特计算机科学系、应用网络安全国家研究中心ATHENE);Data Science Group, University of Hamburg(汉堡大学数据科学组)
Reviewing the Reviewer: LLM-Assisted Reviewer Feedback Generation for Guideline Compliance
Ubiquitous Knowledge Processing Lab (UKP Lab), Department of Computer Science, Technical University of Darmstadt and National Research Center for Applied Cybersecurity ATHENE, Germany(技术大学达姆施塔特计算机科学系 Ubiquitous Knowledge Processing Lab(UKP Lab)及国家应用网络安全研究中心 ATHENE,德国);Department of Data Science & AI, Monash University, Australia(墨尔本大学数据科学与人工智能系,澳大利亚);Data Science Group, University of Hamburg(汉堡大学数据科学组)
RPCBench: A Benchmark for Proactive Premise Critique in LLM-based Recommendation
School of Artificial Intelligence, Jilin University(吉林大学人工智能学院);Engineering Research Center of Knowledge-Driven Human-Machine Intelligence, MOE, China(教育部知识驱动人机智能工程研究中心);International Center of Future Science, Jilin University(吉林大学未来科学国际中心)
LLM-Based Scientific Peer Review: Methods, Benchmarks, and Reliability Challenges
L3S Research Center, Leibniz University Hannover(莱布尼茨汉诺威大学L3S研究中心);Peter L. Reichertz Institute for Medical Informatics of TU Braunschweig and Hannover Medical School(布伦瑞克工业大学与汉诺威医学院彼得·L·赖歇茨医学信息学研究所);Lower Saxony Center for AI and Causal Methods in Medicine (CAIMed)(下萨克森州医学人工智能与因果方法中心(CAIMed))
What Can Natural Language Processing Do for Peer Review?
Technical University of Darmstadt(达姆施塔特工业大学);Mohamed bin Zayed University of Artificial Intelligence(穆罕默德·本·扎耶德人工智能大学);Radboud University Nijmegen(奈梅亨拉德堡德大学);Carnegie Mellon University(卡内基梅隆大学);Allen Institute for Artificial Intelligence(艾伦人工智能研究所);Bocconi University(博科尼大学);The University of Sydney(悉尼大学);Universität Hamburg(汉堡大学);University of British Columbia(不列颠哥伦比亚大学);University of Applied Sciences Darmstadt(达姆施塔特应用科学大学);Université Paris-Saclay(巴黎萨克雷大学);Monash University(莫纳什大学);The Hebrew University of Jerusalem(耶路撒冷希伯来大学);University of Washington(华盛顿大学);Georgia Institute of Technology(佐治亚理工学院);IT University of Copenhagen(哥本哈根信息技术大学)
How Much Were You Told? Measuring External Information in Peer Reviews
Sorbonne Université(索邦大学);CNRS(法国国家科学研究中心);ISIR(智能系统与机器人研究所);International Laboratory on Learning Systems (ILLS)(国际学习系统实验室);Quebec AI Institute (MILA)(魁北克人工智能研究所);CentraleSupélec(巴黎中央理工-高等电力学院);Université Paris-Saclay(巴黎萨克雷大学)
LLM Self-Correction with DeCRIM: Decompose, Critique, and Refine for Enhanced Following of Instructions with Multiple Constraints
Amazon AGI Foundations(亚马逊AGI基础研究院);Télécom Paris(巴黎电信学院);Institut Polytechnique de Paris(巴黎综合理工学院);UNC Chapel Hill(北卡罗来纳大学教堂山分校);University of California, Los Angeles(加利福尼亚大学洛杉矶分校)
Reviewing Clinical Knowledge in Medical Large Language Models: Training and Beyond
School of Computer Science and Technology, Wuhan University of Science and Technology(计算机科学与技术学院,武汉科技大学);Hubei Province Key Laboratory of Intelligent Information Processing and Real-time Industrial System(湖北省智能信息处理与实时工业系统重点实验室);School of Computer Science and Technology, Huazhong University of Science and Technology(计算机科学与技术学院,华中科技大学);School of Cyber Science and Engineering, Wuhan University(网络科学与工程学院,武汉大学);Department of Computer Science, Vrije Universiteit Amsterdam(计算机科学系,阿姆斯特丹自由大学)
AAAR-1.0: Assessing AI's Potential to Assist Research
Netflix(奈飞公司);University of California, Davis(加州大学戴维斯分校);University of Illinois Chicago(伊利诺伊大学芝加哥分校);University of Alabama at Birmingham(阿拉巴马大学伯明翰分校);Ohio State University(俄亥俄州立大学)
Bridge-AI Lab(Bridge-AI实验室);University of Central Florida(中佛罗里达大学);Auburn University(奥本大学);Wichita State University(威奇托州立大学);Gustavus Adolphus College(古斯塔夫斯阿道夫学院)
Beyond Rating: A Comprehensive Evaluation and Benchmark for AI Reviews
College of Computer Science and Artificial Intelligence, Fudan University, Shanghai, China(复旦大学计算机科学与人工智能学院);Shanghai Innovation Institute, Shanghai, China(上海创新研究院)
NLPeer: A Unified Resource for the Computational Study of Peer Review
Ubiquitous Knowledge Processing Lab (UKP Lab)(普适知识处理实验室(UKP实验室));Department of Computer Science(计算机科学系);Hessian Center for AI (hessian.AI)(黑森人工智能中心(hessian.AI));Technical University of Darmstadt(达姆施塔特工业大学)