arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2409.00128cs.CLcs.AIecon.GNq-fin.EC
收藏

大型语言模型能否取代人类受试者?心理学与管理学中基于情景实验的大规模复现研究

Can Large Language Models Replace Human Subjects? A Large-Scale Replication of Scenario-Based Experiments in Psychology and Management

Tsinghua University(清华大学)

Ziyan Cui, Ning Li, Huaikang Zhou

AI总结:

本研究通过复现156项心理学实验,发现LLMs虽能高效复现主效应和交互效应,但效应量偏高,对敏感话题复现率低,可作为辅助工具而非替代人类受试者。

摘要:

人工智能(AI)正日益融入科学研究,尤其是在理解人类行为至关重要的社会科学领域。大型语言模型(LLMs)在复现各种心理学实验中的人类类似反应方面已展现出潜力。我们开展了一项大规模研究,使用三种最先进的LLMs(GPT-4、Claude 3.5 Sonnet和DeepSeek v3)复现了来自顶级社会科学期刊的156项心理学实验。我们的结果显示,虽然LLMs对主效应表现出较高的复现率(73-81%),对交互效应的复现成功率从中等到较强(46-63%),但它们产生效应量始终大于人类研究,Fisher Z值约为人类研究的2-3倍。值得注意的是,LLMs在涉及种族、性别和伦理等社会敏感话题的研究中复现率显著较低。当原始研究报告零结果时,LLMs以极高的比率(68-83%)产生显著结果——虽然这可能反映了更干净的数据和更少的噪声(如更窄的置信区间所示),但也暗示了效应量被高估的潜在风险。我们的结果展示了LLMs在心理学研究中的前景与挑战,为预测试和快速假设验证提供了高效工具,同时丰富而非取代传统的人类受试者研究,但对于复杂社会现象和文化敏感的研究问题,需要更细致的解读和人类验证。

英文摘要:

Artificial Intelligence (AI) is increasingly being integrated into scientific research, particularly in the social sciences, where understanding human behavior is critical. Large Language Models (LLMs) have shown promise in replicating human-like responses in various psychological experiments. We conducted a large-scale study replicating 156 psychological experiments from top social science journals using three state-of-the-art LLMs (GPT-4, Claude 3.5 Sonnet, and DeepSeek v3). Our results reveal that while LLMs demonstrate high replication rates for main effects (73-81%) and moderate to strong success with interaction effects (46-63%), They consistently produce larger effect sizes than human studies, with Fisher Z values approximately 2-3 times higher than human studies. Notably, LLMs show significantly lower replication rates for studies involving socially sensitive topics such as race, gender and ethics. When original studies reported null findings, LLMs produced significant results at remarkably high rates (68-83%) - while this could reflect cleaner data with less noise, as evidenced by narrower confidence intervals, it also suggests potential risks of effect size overestimation. Our results demonstrate both the promise and challenges of LLMs in psychological research, offering efficient tools for pilot testing and rapid hypothesis validation while enriching rather than replacing traditional human subject studies, yet requiring more nuanced interpretation and human validation for complex social phenomena and culturally sensitive research questions.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑