Automated reproducibility assessments in the social and behavioral sciences using large language models
LMU Munich(慕尼黑大学);Munich Center for Machine Learning(慕尼黑机器学习中心);University of Cologne(科隆大学)
Tobias Holtdirk, Pietro Marcolongo, Anna Steinberg Schulten, Felix Henninger, Stefan Rose, Sarah Ball, Bolei Ma, Frauke Kreuter, Markus Weinmann, Stefan Feuerriegel
Reproducibility in the social and behavioral sciences is typically evaluated by independent researchers who reanalyze the original data to assess whether the published findings can be recovered. However, such approaches are resource-intensive and difficult to scale. Here, we show that large language models (LLMs) can automate reproducibility assessments. Using N = 180 published studies with predefined claims from the behavioral and social sciences, we compare LLM-generated analyses with the original findings. For 11 studies, the LLM pipeline could not produce a viable effect size estimate. For the remaining studies, the LLM reached the same qualitative conclusion as the original study in 80% of cases, and recovered the original effect sizes (using a +/-0.05 tolerance in Cohen's d) in 24% of studies. In a subset with human reanalyses, the LLM reached the same qualitative conclusion as the original study in 95% of studies, similar to human reanalysts (83%), and the LLM recovered the original effect sizes using a +/-0.05 tolerance in 40% of studies, again broadly similar to human reanalysts (28%). Given the current capabilities and limitations of LLMs, the findings show that LLMs can support systematic audits of empirical results rather than substitute expert judgment. As such, LLMs can serve as a scalable screening tool to improve the rigor and reproducibility in empirical research.
Systematic Exploration of Dialogue Summarization Approaches for Reproducibility, Comparative Assessment, and Methodological Innovations for Advancing Natural Language Processing in Abstractive Summarization
SRM Institute of Science and Technology(SRM科学技术学院);SRM University, AP(SRM大学(安得拉邦))
Unveiling the Landscape of Clinical Depression Assessment: From Behavioral Signatures to Psychiatric Reasoning
School of Computer Science and Engineering, Central South University(中南大学计算机科学与工程学院);CoAI Group, DCST, IAI, BNRIST, Tsinghua University(清华大学);University of International Relations(国际关系大学);Central China Normal University(华中师范大学);Lingxin AI(灵心AI);Yuquan Hospital, Tsinghua University(清华大学友谊医院)
LLMs can Perform Multi-Dimensional Analytic Writing Assessments: A Case Study of L2 Graduate-Level Academic English Writing
Department of Linguistics & Institute for Advanced Computational Science, Stony Brook University(语言学系与高级计算科学研究所,石溪大学);Department of Linguistics, University of Saskatchewan(语言学系,萨斯喀彻温大学)
Large Language Model-Based Agents for Automated Research Reproducibility: An Exploratory Study in Alzheimer's Disease
Biomedical Informatics & Data Science, Johns Hopkins University(约翰霍普金斯大学生物医学信息学与数据科学);Biomedical & Health Informatics, University of Washington(华盛顿大学生物医学与健康信息学)
AgentBeats: Agentifying Agent Assessment for Openness, Standardization, and Reproducibility
University of California, Berkeley(加州大学伯克利分校);Purdue University(普渡大学);University of Ljubljana(卢布尔雅那大学);University of Washington(华盛顿大学);Oasis Labs;University of Maryland(马里兰大学);IBM Research(IBM研究院);Mila;McGill University(麦吉尔大学);ServiceNow Research(ServiceNow研究院);Carnegie Mellon University(卡内基梅隆大学);National Institute of Standards and Technology(美国国家标准与技术研究院);The Ohio State University(俄亥俄州立大学);University of Cambridge(剑桥大学);University of California, Santa Barbara(加州大学圣塔芭芭拉分校)
The largest EEG-based BCI reproducibility study for open science: the MOABB benchmark
Inria(法国国家信息与自动化研究所);Université Paris-Saclay(巴黎-萨克雷大学);Donders Institute(唐德斯研究所);Radboud University(拉德堡德大学);University of São Paulo(圣保罗大学);Federal University of ABC(ABC联邦大学);GeorgiaTech-CNRS IRL 2958(佐治亚理工学院-法国国家科学研究中心国际联合实验室2958);Centralesupelec(中央理工-高等电力学院);CEA(法国原子能和替代能源委员会)