通过同行评审治理人工智能研究：关于伦理标记在重新提交中纵向效应的混合方法研究

中文摘要 AI 辅助

选择性人工智能会议最近开始强制执行伦理标记及相关评审要求，其目标是在论文发表前引导研究走向更安全、更负责任的做法。但这些要求是否真的按预期引导了研究？在本文中，我们表明，作者在收到伦理标记后，更常修改项目的呈现方式，而非调整其潜在的研究议程。我们首先通过追踪在ICLR评审中收到伦理标记的被拒和撤回投稿在后续公开重新提交中的情况，研究伦理标记的纵向效应，跟踪ICLR评审结束（即原始评审者不再监督项目）后的稿件变化。我们根据评审后的变化将这些重新提交定性编码为五个类别，发现在446个案例中有83%的情况下，作者未解决被标记的担忧，或修改论文但未改变所涉及的方法或程序。然后我们提问：如果作者很少因伦理标记而改变研究，那他们改变了什么？为回答此问题，我们手动阅读了25个案例的评审和反驳意见，并直接采访作者关于其反驳过程和重新提交决策。我们发现，在评审者可以更新其评估时，作者常在反驳中承认担忧，但在被拒后，当他们不认为批评是改变研究的合理理由时，会放弃这些承认。受访者将出版物的变化与研究方向的改变分开描述，称评审为“编辑过程”，塑造“哪些故事被看到”，在另一案例中，称同行评审“主要是为了筛选论文”。作者更愿意改变他们发表的内容，而非他们研究或构建的内容；因此，我们建议政策变更，尤其是在重新提交时披露先前的伦理标记，以便问责得以延续。

英文摘要

Selective AI conferences have recently begun enforcing ethics flags and related review requirements, with the goal being to steer research towards safer and more responsible practices before publication. But do these requirements actually steer research as intended? In this paper, we show that authors more often revise how projects are presented following ethics flags than redirect their underlying research agendas. We first study the longitudinal effects of ethics flags by following rejected and withdrawn ICLR submissions with ethics flags into later public resubmissions, tracking manuscript changes after the ICLR review ends, when the original reviewers no longer oversee the project. We qualitatively code these resubmissions into five categories based on what changed after review and find that in 83% of 446 cases, authors leave the flagged concern unaddressed or revise the paper without changing the implicated methods or procedures. Then, we ask: if authors rarely change the research in response to ethics flags, what do they change instead? To answer this, we manually read reviews and rebuttals from 25 cases and directly interview authors about their rebuttal processes and resubmission decisions. We find that authors often concede concerns during rebuttal when reviewers can update their assessments, but drop those concessions after rejection when they do not regard the criticism as a sound reason to change the research. Interview participants describe publication changes as separate from changes to research direction, calling review an "editorial process" that shapes "what stories get seen" and, in another case, saying peer reviews are "mostly to filter out papers." Authors more readily change what they publish than what they study or build; we therefore recommend policy changes, especially disclosure of prior ethics flags upon resubmission so accountability carries over.

发表机构

• Massachusetts Institute of Technology（麻省理工学院）
• Harvard University（哈佛大学）

机构由 AI 辅助整理，请以论文原文为准。