多智能体辩论能否改善人工智能对研究论文的反馈？
Does Multi-Agent Debate Improve AI Feedback on Research Papers?
浏览论文内容
中文总结 AI 辅助
研究探讨多智能体辩论能否改善人工智能对研究论文的反馈。通过44篇元分析作者对三篇人工智能报告排名，发现作者更青睐前沿模型单次审阅，且不同评判排序有差异，警示不能用人工智能评判替代作者，还衡量了对已完成论文的有用性。
中文摘要 AI 辅助
可能不行，至少在经济学的元分析中是这样。在一项预先注册、身份掩码的论文内实验中，44篇元分析的作者根据对改进论文的有用性对三篇关于其论文的人工智能报告进行排名：前沿模型的单次审阅与我们构建的两个多智能体辩论工具进行比较，我们原本预计后者会胜出。所有报告都有相同的长度和模板。作者更喜欢单次审阅，比mad - research高出0.66个排名点（95%置信区间为0.32至1.00），比paper - workshop高出0.57个排名点（0.16至0.95），尽管paper - workshop花费的令牌大约是三十倍。记得期刊审稿人报告的作者通常将其排在首位且从未排在最后；在另一项练习中，三位人工智能评判几乎总是将真实的期刊审稿人报告排在最后。在这三篇人工智能报告中，Gemini（其模型家族未撰写任何一份报告中的评判）会在作者的位置上将paper - workshop排在首位，这与单次审阅的偏好相反。这种反转警示不要用人工智能评判替代作者。我们衡量了对已完成论文感知到的有用性；人工智能是否应该审阅论文是另一个问题。
英文摘要
Probably not, at least for meta-analyses in economics. In a pre-registered, identity-masked, within-paper experiment, the authors of 44 meta-analyses ranked three AI reports on their own paper by usefulness for improving it: a single pass by a frontier model against two multi-agent debate tools we built and expected to win. All reports were held to a common length and template. The authors preferred the single pass, by 0.66 rank points over mad-research (95% CI 0.32 to 1.00) and 0.57 over paper-workshop (0.16 to 0.95), though paper-workshop spent roughly thirty times the tokens. Authors who recalled their journal referee report usually placed it first and never last; in a separate exercise, three AI judges almost always placed the real journal referee report last. Among the three AI reports, Gemini (the judge whose model family wrote none of the reports) would have ranked paper-workshop first in the authors' place, reversing the single-pass preference. The reversal warns against substituting an AI judge for the author. We measure perceived usefulness for finished papers; whether AI should referee papers is a separate question.
发表机构
- Charles University, Faculty of Social Sciences, Institute of Economic Studies（查理大学社会科学学院经济研究所）
- Charles University, Prague Centre for Economic Policy Research, London（查理大学布拉格经济政策研究中心，伦敦）
- Meta-Research Innovation Center at Stanford（斯坦福大学元研究创新中心）
机构由 AI 辅助整理，请以论文原文为准。
