arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2606.13670cs.AI
收藏

使用大型语言模型自动评估社会与行为科学的可重复性

Automated reproducibility assessments in the social and behavioral sciences using large language models

LMU Munich(慕尼黑大学);Munich Center for Machine Learning(慕尼黑机器学习中心);University of Cologne(科隆大学)

Tobias Holtdirk, Pietro Marcolongo, Anna Steinberg Schulten, Felix Henninger, Stefan Rose, Sarah Ball, Bolei Ma, Frauke Kreuter, Markus Weinmann, Stefan Feuerriegel

AI总结:

本研究利用大型语言模型(LLMs)自动评估社会与行为科学研究的可重复性,在76项研究中,LLM在41%的研究中恢复了原始效应量,在96%的案例中得出了与原始研究相同的定性结论,优于人类再分析。

摘要:

社会与行为科学的可重复性通常由独立研究人员重新分析原始数据来评估,以判断已发表的研究结果是否可复现。然而,这种方法资源密集且难以规模化。在此,我们展示了大型语言模型(LLMs)可以自动化可重复性评估。利用N=76项来自行为与社会科学、具有预定义声明的研究,我们比较了LLM生成的分析与原始结果和人类再分析。对于7项研究,LLM无法产生可行的效应量估计。对于其余研究,我们的LLM流程在41%的研究中恢复了原始效应量(Cohen's d的容忍度为+/-0.05)。此外,我们的LLM流程在96%的案例中得出了与原始研究相同的定性结论,其中结论指示再分析是否支持原始声明。相比之下,人类再分析者在34%的研究中恢复了原始效应量,并在74%的案例中得出了相同的定性结论。这些结果共同表明,LLMs可以作为自动化可重复性评估的可扩展工具,并为社会与行为科学中实证结果的系统审计提供基础。

英文摘要:

Reproducibility in the social and behavioral sciences is typically evaluated by independent researchers who reanalyze the original data to assess whether the published findings can be recovered. However, such approaches are resource-intensive and difficult to scale. Here, we show that large language models (LLMs) can automate reproducibility assessments. Using N = 180 published studies with predefined claims from the behavioral and social sciences, we compare LLM-generated analyses with the original findings. For 11 studies, the LLM pipeline could not produce a viable effect size estimate. For the remaining studies, the LLM reached the same qualitative conclusion as the original study in 80% of cases, and recovered the original effect sizes (using a +/-0.05 tolerance in Cohen's d) in 24% of studies. In a subset with human reanalyses, the LLM reached the same qualitative conclusion as the original study in 95% of studies, similar to human reanalysts (83%), and the LLM recovered the original effect sizes using a +/-0.05 tolerance in 40% of studies, again broadly similar to human reanalysts (28%). Given the current capabilities and limitations of LLMs, the findings show that LLMs can support systematic audits of empirical results rather than substitute expert judgment. As such, LLMs can serve as a scalable screening tool to improve the rigor and reproducibility in empirical research.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑