LLM作为科学论文作者清单助手的实用性：NeurIPS'24实验
Usefulness of LLMs as an Author Checklist Assistant for Scientific Papers: NeurIPS'24 Experiment
AI总结：
本研究在NeurIPS'24会议上实验评估了LLM作为论文清单助手的实用性，发现超70%作者认为有用并会据其反馈修改，但也存在不准确、过度严格及可被操纵的漏洞。
摘要：
大型语言模型（LLM）在辅助科学同行评审方面是一种有前景但有争议的工具。本研究评估了LLM在会议环境中作为审查论文是否符合投稿标准的工具的实用性。我们在2024年神经信息处理系统（NeurIPS）会议上进行了一项实验，234篇论文被自愿提交给一个“基于LLM的清单助手”。该助手验证论文是否符合NeurIPS使用的作者清单，该清单包含确保符合研究和手稿准备标准的问题。NeurIPS论文作者对助手的评估表明，基于LLM的助手在验证清单完成情况方面总体上是有帮助的。在使用后的调查中，超过70%的作者认为该助手有用，70%的作者表示他们会根据其反馈修改论文或清单回复。虽然对助手的因果归因并非确定性的，但定性证据表明LLM有助于改进一些投稿。调查回复和重新提交的分析表明，作者根据LLM的具体反馈对其投稿进行了实质性修改。该实验还突出了LLM的常见问题：不准确性（20/52）和过度严格（14/52）是作者标记的最频繁问题。我们还进行了实验以了解可能对系统进行博弈的情况，结果显示该助手可能通过捏造的正当理由被操纵以提高分数，这凸显了自动评审工具的潜在漏洞。
英文摘要：
Large language models (LLMs) represent a promising, but controversial, tool in aiding scientific peer review. This study evaluates the usefulness of LLMs in a conference setting as a tool for vetting paper submissions against submission standards. We conduct an experiment at the 2024 Neural Information Processing Systems (NeurIPS) conference, where 234 papers were voluntarily submitted to an "LLM-based Checklist Assistant." This assistant validates whether papers adhere to the author checklist used by NeurIPS, which includes questions to ensure compliance with research and manuscript preparation standards. Evaluation of the assistant by NeurIPS paper authors suggests that the LLM-based assistant was generally helpful in verifying checklist completion. In post-usage surveys, over 70% of authors found the assistant useful, and 70% indicate that they would revise their papers or checklist responses based on its feedback. While causal attribution to the assistant is not definitive, qualitative evidence suggests that the LLM contributed to improving some submissions. Survey responses and analysis of re-submissions indicate that authors made substantive revisions to their submissions in response to specific feedback from the LLM. The experiment also highlights common issues with LLMs: inaccuracy (20/52) and excessive strictness (14/52) were the most frequent issues flagged by authors. We also conduct experiments to understand potential gaming of the system, which reveal that the assistant could be manipulated to enhance scores through fabricated justifications, highlighting potential vulnerabilities of automated review tools.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
