arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2402.03530cs.HC
收藏

ReviewFlow:支持学术同行评审的智能脚手架

ReviewFlow: Intelligent Scaffolding to Support Academic Peer Reviewing

Lu Sun, Aaron Chan, Yun Seo Chang, Steven P. Dow

AI总结:

针对新手同行评审缺乏指导的问题,本文提出AI驱动工作流ReviewFlow,通过情境反思、知识支持和笔记综合帮助新手撰写更全面的评审,实验表明其能识别更多优缺点。

摘要:

同行评审是科学的基石。研究社区通过同行评审来评估贡献并提升科学工作的整体质量。每年都有新的社区成员首次被招募为同行评审人。技术如何帮助新手遵循其社区的同行评审实践与标准?为了更好地理解同行评审实践与挑战,我们对10名新手和10名专家进行了形成性研究。我们发现,许多专家采用一种工作流程:先进行标注、记笔记,再将笔记综合成符合社区标准、论证充分的评审意见。新手在如何阅读和评估投稿以及如何组织论文评审结构方面缺乏及时指导。为了支持同行评审过程,我们开发了ReviewFlow——一种AI驱动的工作流程,通过情境化反思来脚手架式地引导新手批判和标注投稿,提供情境知识支持以评估新颖性,并通过笔记到大纲的综合来帮助使同行评审符合社区期望。在一项被试内实验中,16名缺乏经验的评审人在两种条件下撰写评审:使用ReviewFlow和使用仅提供最少指导的基线环境。使用ReviewFlow时,参与者产出了更全面的评审,识别出更多的优点和缺点。虽然参与者赞赏ReviewFlow提供的流程化支持,但他们也对在科学评审过程中使用AI表达了担忧。我们讨论了利用AI为同行评审过程提供脚手架对科学工作及其他领域的影响。

英文摘要:

Peer review is a cornerstone of science. Research communities conduct peer reviews to assess contributions and to improve the overall quality of science work. Every year, new community members are recruited as peer reviewers for the first time. How could technology help novices adhere to their community's practices and standards for peer reviewing? To better understand peer review practices and challenges, we conducted a formative study with 10 novices and 10 experts. We found that many experts adopt a workflow of annotating, note-taking, and synthesizing notes into well-justified reviews that align with community standards. Novices lack timely guidance on how to read and assess submissions and how to structure paper reviews. To support the peer review process, we developed ReviewFlow -- an AI-driven workflow that scaffolds novices with contextual reflections to critique and annotate submissions, in-situ knowledge support to assess novelty, and notes-to-outline synthesis to help align peer reviews with community expectations. In a within-subjects experiment, 16 inexperienced reviewers wrote reviews in two conditions: using ReviewFlow and using a baseline environment with minimal guidance. With ReviewFlow, participants produced more comprehensive reviews, identifying more pros and cons. While participants appreciated the streamlined process support from ReviewFlow, they also expressed concerns about using AI as part of the scientific review process. We discuss the implications of using AI to scaffold the peer review process on scientific work and beyond.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑