arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2605.16623cs.CYcs.AI
收藏

信任还是不信任:作者对基于AI的评论的回应

To Trust or Not to Trust: Authors' Response to AI-based Reviews

École Normale Supérieure(巴黎高等师范学院);Sorbonne University(索邦大学);University of West Bohemia(西波什埃大学);Massachusetts Institute of Technology(麻省理工学院)

César Leblanc, Lukas Picek

AI总结:

本文通过两项研究探讨作者对AI辅助评审的使用和看法,发现大多数作者认为AI反馈有用,但不将其等同于人类评审,且更倾向于在提交前使用AI作为内部工具。

摘要:

大型语言模型日益被讨论和用于协助学术同行评审,但关于作者如何使用和感知AI反馈的实证证据仍有限。本文报告了两项独立试点研究的结果,研究了作者在两个计算机科学会议中对AI辅助评审的使用和看法。在评审发布后,作者被邀请完成一份匿名的评审后问卷,询问AI评审的有用性、可信度、与人类评审的一致性、修改的实用性、感知的不准确性以及同意。最终的数据集包含40篇论文作者的56个可分析响应;封闭式问题使用描述性统计汇总,开放式回答使用归纳主题分析。大多数受访者(83.9%)认为AI评审有用,80.4%报告说AI发现了人类评审未提及的问题。这种感知的附加价值转化为行动:82.1%的受访者报告在最终版本中使用了至少一些AI反馈。然而,作者并不将AI评审视为等同于人类评审。他们普遍认为AI的可信度低于人类评审,尽管25.0%的受访者描述至少一些人类评审不很有用。报告的AI评审问题通常有限:51.8%报告了轻微的不准确,而16.1%报告了明显错误、误导或不相关的评论。对未来发展使用的支持最强当AI被框架为监督或作者控制的工具:96.4%表示在未来的提交中会使用AI作为内部评审工具,89.3%更倾向于提前得知AI将在评审中使用,76.8%更倾向于在使用前获得明确的同意。

英文摘要:

Large language models are increasingly discussed and used as tools that may assist with scholarly peer review, but empirical evidence regarding how authors use and perceive AI-based feedback remains limited. This paper reports findings from two independent pilot studies on authors' use and perceptions of AI-based auxiliary review at two computer science venues. After the review release, authors were invited to complete an anonymous post-review questionnaire about the AI review's usefulness, trustworthiness, agreement with human reviews, practical value for revision, perceived inaccuracies, and consent. The final dataset included 56 analyzable responses from authors of 40 papers; closed-ended items were summarized using descriptive statistics, and open-ended responses were analyzed using inductive thematic analysis. Most respondents (83.9%) considered the AI-based review useful, and 80.4% reported that it identified issues not mentioned by human reviewers. This perceived added value translated into action: 82.1% reported using at least some AI feedback in their camera-ready version. However, the authors did not treat the AI review as equivalent to a human review. They generally trusted it less than the human reviews and found human feedback clearer, even though 25.0% described at least some human reviews as not very useful. Reported problems with the AI review were usually limited: 51.8% reported minor inaccuracies, while 16.1% reported clearly incorrect, misleading, or irrelevant comments. Support for future use was strongest when AI was framed as a supervised or author-controlled tool: 96.4% said they would use AI as an internal review tool before future submissions, 89.3% preferred advance notice that AI would be used in review, and 76.8% favored explicit consent before use.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑