arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2604.03338econ.GNcs.AIcs.CYq-fin.EC
收藏

想法瓶颈:分解人工智能生成与人类经济学研究之间的质量差距

The Ideation Bottleneck: Decomposing the Quality Gap Between AI-Generated and Human Economics Research

Tsinghua University(清华大学);Social Catalyst Lab at the University of Zurich(苏黎世大学社会催化剂实验室)

Ning Li

AI总结:

本文通过分析953篇经济学论文,发现人工智能生成的研究想法质量显著低于人类,占整体质量差异的71%,执行质量差距较小,但机制分析深度不足仍是主要瓶颈。

摘要:

自主AI系统现在能够生成完整的经济学研究论文,但在直接比较中显著劣于人类撰写的论文。本文将质量差距分解为两个独立组成部分:研究想法质量和执行质量。使用经过出版决策训练的微调语言模型(Gong, Li, and Zhou, 2026)评估想法质量,以及由Gemini 3.1 Flash Lite评估的六维评分标准评估执行质量,分析了953篇经济学论文——912篇由APE项目生成的AI论文和41篇发表在《美国经济评论》和《经济政策》上的人类论文。想法质量差距较大(Cohen's d = 2.23, p < 0.001),人类论文的平均群体异常概率为47.1%,而AI论文为16.5%。执行质量差距也显著但较小(d = 0.90, p < 0.001),人类论文得分为4.38/5.0,而AI论文得分为3.84。想法质量占整体质量差异的约71%,执行质量占29%。最大的执行弱点是机制分析深度(d = 1.43);在稳健性上没有显著差异。我们发现74%的AI论文使用差分-in-差分,只有7篇AI论文(0.8%)在想法和执行质量上同时超过中位数的人类论文。人工智能生成经济学研究的主要瓶颈仍然是想法生成。

英文摘要:

Autonomous AI systems can now generate complete economics research papers, but they substantially underperform human-authored publications in head-to-head comparisons. This paper decomposes the quality gap into two independent components: research idea quality and execution quality. Using a two-model ensemble of fine-tuned language models trained on publication decisions (Gong, Li, and Zhou, 2026) to evaluate idea quality and a comprehensive six-dimension rubric assessed by Gemini 3.1 Flash Lite -- the same model family used as the APE tournament judge, ensuring methodological consistency -- to evaluate execution quality, we analyze 953 economics papers -- 912 AI-generated papers from the APE project and 41 human papers published in the American Economic Review and AEJ: Economic Policy. The idea quality gap is large (Cohen's d = 2.23, p < 0.001), with human papers achieving 47.1% mean ensemble exceptional probability versus 16.5% for AI. The execution quality gap is also significant but smaller (d = 0.90, p < 0.001), with human papers scoring 4.38/5.0 versus 3.84. Idea quality accounts for approximately 71% of the overall quality difference, with execution contributing 29%. The largest execution weakness is mechanism analysis depth (d = 1.43); no significant difference is found on robustness. We document that 74% of AI papers employ difference-in-differences, and only 7 AI papers (0.8%) surpass the median human paper on both idea and execution quality simultaneously. The primary bottleneck to competitive AI-generated economics research remains ideation.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑