当评审员收到AI反馈时会发生什么?
What happens when reviewers receive AI feedback in their reviews?
AI总结:
研究探讨AI反馈工具在同行评审中的影响,通过实证分析揭示其在提升质量与保护人类责任之间的平衡。
摘要:
AI正在重塑学术研究,但其在同行评审中的作用仍然存在分歧和争议。支持者认为其有潜力减少评审员负担并提高质量,而批评者则警告可能会损害公平性、问责制和信任。在ICLR 2025上,一个官方的AI反馈工具被部署,为评审员提供评审后的建议。我们通过调查和访谈研究了该部署,调查了评审员如何与该工具互动以及他们对工具的可用性和影响的看法。我们的发现揭示了当AI增强同行评审时的机会和紧张局势。这项工作为在实时评审过程中使用此类AI工具提供了首次实证证据,记录了评审员在高风险评审背景下如何回应AI生成的反馈。我们进一步提出了旨在提高质量同时保护人类专业知识、自主权和责任的AI辅助评审设计建议。
英文摘要:
AI is reshaping academic research, yet its role in peer review remains polarising and contentious. Advocates see its potential to reduce reviewer burden and improve quality, while critics warn of risks to fairness, accountability, and trust. At ICLR 2025, an official AI feedback tool was deployed to provide reviewers with post-review suggestions. We studied this deployment through surveys and interviews, investigating how reviewers engaged with the tool and perceived its usability and impact. Our findings surface both opportunities and tensions when AI augments in peer review. This work contributes the first empirical evidence of such an AI tool in a live review process, documenting how reviewers respond to AI-generated feedback in a high-stakes review context. We further offer design implications for AI-assisted reviewing that aim to enhance quality while safeguarding human expertise, agency, and responsibility.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。