arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2406.16253cs.CL
收藏

大语言模型辅助NLP研究者:批判性论文(元)评审

LLMs Assist NLP Researchers: Critique Paper (Meta-)Reviewing

University of Illinois Chicago(伊利诺伊大学芝加哥分校);Salesforce Research(Salesforce研究院);Hong Kong Polytechnic University(香港理工大学);Penn State University(宾夕法尼亚州立大学);Google DeepMind(谷歌DeepMind);University of Southern California(南加州大学);University of California, Davis(加州大学戴维斯分校);University of Basel(巴塞尔大学);Xiaomi AI Lab(小米AI实验室);Scale AI;HKUST(香港科技大学);Georgia Institute of Technology(佐治亚理工学院);University of California, Santa Cruz(加州大学圣克鲁兹分校);Singapore University of Technology and Design(新加坡科技设计大学);Texas A&M University(德克萨斯农工大学);Massey University(梅西大学);University of Liverpool(利物浦大学)
Jiangshu Du, Yibo Wang, Wenting Zhao, Zhongfen Deng, Shuaiqi Liu, Renze Lou, Henry Peng Zou, Pranav Narayanan Venkit, Nan Zhang, Mukund Srinath, Haoran Ranran Z… 展开作者

Jiangshu Du, Yibo Wang, Wenting Zhao, Zhongfen Deng, Shuaiqi Liu, Renze Lou, Henry Peng Zou, Pranav Narayanan Venkit, Nan Zhang, Mukund Srinath, Haoran Ranran Zhang, Vipul Gupta, Yinghui Li, Tao Li, Fei Wang, Qin Liu, Tianlin Liu, Pengzhi Gao, Congying Xia, Chen Xing, Jiayang Cheng, Zhaowei Wang, Ying Su, Raj Sanjay Shah, Ruohao Guo, Jing Gu, Haoran Li, Kangda Wei, Zihao Wang, Lu Cheng, Surangika Ranathunga, Meng Fang, Jie Fu, Fei Liu, Ruihong Huang, Eduardo Blanco, Yixin Cao, Rui Zhang, Philip S. Yu, Wenpeng Yin

AI总结:

本文构建ReviewCritique数据集,系统评估LLM在论文评审和元评审中的表现,比较其生成评审与人工评审的质量差异,并考察LLM识别评审缺陷的能力。

摘要:

本工作受到两个关键趋势的驱动。一方面,大语言模型(LLMs)在写作、绘画和问答等多种生成任务中展现出卓越的多功能性,显著减少了许多日常任务所需的时间。另一方面,研究人员的工作不仅耗时,而且对专业知识要求极高,他们面临着日益严峻的挑战,因为必须花费更多时间阅读、撰写和评审论文。这引出了一个问题:LLMs如何能够潜在地帮助研究人员减轻繁重的工作负担?本研究聚焦于“LLMs辅助NLP研究者”这一主题,特别考察LLM在辅助论文(元)评审方面的有效性及其可识别性。为此,我们构建了ReviewCritique数据集,其中包含两类信息:(i)NLP论文(初始投稿版本而非最终定稿版本),同时附带人工撰写的评审和LLM生成的评审;(ii)每篇评审都带有由专家标注的“缺陷”标签,以及针对各个片段的相应解释。利用ReviewCritique,本研究探索了两条研究问题线索:(i)“LLMs作为评审者”,LLM生成的评审与人工撰写的评审在质量和可区分性方面相比如何?(ii)“LLMs作为元评审者”,LLM能在多大程度上有效识别单篇论文评审中的潜在问题,例如有缺陷或不专业的评审片段?据我们所知,这是首项提供如此全面分析的工作。

英文摘要:

This work is motivated by two key trends. On one hand, large language models (LLMs) have shown remarkable versatility in various generative tasks such as writing, drawing, and question answering, significantly reducing the time required for many routine tasks. On the other hand, researchers, whose work is not only time-consuming but also highly expertise-demanding, face increasing challenges as they have to spend more time reading, writing, and reviewing papers. This raises the question: how can LLMs potentially assist researchers in alleviating their heavy workload? This study focuses on the topic of LLMs assist NLP Researchers, particularly examining the effectiveness of LLM in assisting paper (meta-)reviewing and its recognizability. To address this, we constructed the ReviewCritique dataset, which includes two types of information: (i) NLP papers (initial submissions rather than camera-ready) with both human-written and LLM-generated reviews, and (ii) each review comes with "deficiency" labels and corresponding explanations for individual segments, annotated by experts. Using ReviewCritique, this study explores two threads of research questions: (i) "LLMs as Reviewers", how do reviews generated by LLMs compare with those written by humans in terms of quality and distinguishability? (ii) "LLMs as Metareviewers", how effectively can LLMs identify potential issues, such as Deficient or unprofessional review segments, within individual paper reviews? To our knowledge, this is the first work to provide such a comprehensive analysis.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑