arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2408.09682cs.AI
收藏

使用大语言模型模拟现场实验

Simulating Field Experiments with Large Language Models

University of Illinois at Chicago(伊利诺伊大学芝加哥分校)

Yaoyu Chen, Yuheng Hu, Yingda Lu

AI总结:

本研究提出观察者与参与者两种提示策略,首次用大语言模型模拟现场实验,在特定场景下准确率达66%,同时明确了模型在性别差异等主题上的能力边界,为预实验辅助及大规模论文筛选提供了可能。

摘要:

主流大语言模型(LLMs)凭借其前所未有的内容生成与推理能力,能够模拟人类的应答。然而,目前尚不清楚是否以及如何利用LLMs模拟现场实验。本文提出并评估了两种提示策略:可直接预测主要结论的观察者模式,以及模拟参与者应答分布的参与者模式。\n\n我们采用该方法检验了15篇发表于INFORMS和MISQ的高被引现场实验论文,发现在特定场景下,模拟实验结果与实际结果之间存在令人鼓舞的一致性。我们进一步明确了LLMs表现不佳的主题,包括性别差异和社会规范相关研究。此外,本文提出的自动化、标准化工作流程为大规模筛选更多包含现场实验的论文提供了可能。\n\n本文率先将大语言模型(LLMs)应用于现场实验模拟,是对以往仅聚焦于实验室环境的相关研究的重要拓展。通过引入观察者模式和参与者模式这两种新型提示策略,我们证明了LLMs在复杂现场场景中既能够预测实验结果,也能够复现参与者的应答。研究结果显示,LLMs在特定场景下的模拟结果与实际实验结果具有可观的一致性,其中观察者模式的模拟准确率达到66%。本研究拓展了LLMs的潜在应用范围,阐明了其在研究人员开展成本高昂的现场实验前提供辅助的实用价值。此外,本研究还明确了LLMs用于模拟现场实验时的能力边界,为考虑将LLMs纳入实验工具库的研究人员提供了警示。

英文摘要:

Prevailing large language models (LLMs) are capable of human responses simulation through its unprecedented content generation and reasoning abilities. However, it is not clear whether and how to leverage LLMs to simulate field experiments. In this paper, we propose and evaluate two prompting strategies: the observer mode that allows a direct prediction on main conclusions and the participant mode that simulates distributions of responses from participants. Using this approach, we examine fifteen well cited field experimental papers published in INFORMS and MISQ, finding encouraging alignments between simulated experimental results and the actual results in certain scenarios. We further identify topics of which LLMs underperform, including gender difference and social norms related research. Additionally, the automatic and standardized workflow proposed in this paper enables the possibility of a large-scale screening of more papers with field experiments. This paper pioneers the utilization of large language models (LLMs) for simulating field experiments, presenting a significant extension to previous work which focused solely on lab environments. By introducing two novel prompting strategies, observer and participant modes, we demonstrate the ability of LLMs to both predict outcomes and replicate participant responses within complex field settings. Our findings indicate a promising alignment with actual experimental results in certain scenarios, achieving a stimulation accuracy of 66% in observer mode. This study expands the scope of potential applications for LLMs and illustrates their utility in assisting researchers prior to engaging in expensive field experiments. Moreover, it sheds light on the boundaries of LLMs when used in simulating field experiments, serving as a cautionary note for researchers considering the integration of LLMs into their experimental toolkit.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑