Prevailing large language models (LLMs) are capable of human responses simulation through its unprecedented content generation and reasoning abilities. However, it is not clear whether and how to leverage LLMs to simulate field experiments. In this paper, we propose and evaluate two prompting strategies: the observer mode that allows a direct prediction on main conclusions and the participant mode that simulates distributions of responses from participants. Using this approach, we examine fifteen well cited field experimental papers published in INFORMS and MISQ, finding encouraging alignments between simulated experimental results and the actual results in certain scenarios. We further identify topics of which LLMs underperform, including gender difference and social norms related research. Additionally, the automatic and standardized workflow proposed in this paper enables the possibility of a large-scale screening of more papers with field experiments. This paper pioneers the utilization of large language models (LLMs) for simulating field experiments, presenting a significant extension to previous work which focused solely on lab environments. By introducing two novel prompting strategies, observer and participant modes, we demonstrate the ability of LLMs to both predict outcomes and replicate participant responses within complex field settings. Our findings indicate a promising alignment with actual experimental results in certain scenarios, achieving a stimulation accuracy of 66% in observer mode. This study expands the scope of potential applications for LLMs and illustrates their utility in assisting researchers prior to engaging in expensive field experiments. Moreover, it sheds light on the boundaries of LLMs when used in simulating field experiments, serving as a cautionary note for researchers considering the integration of LLMs into their experimental toolkit.
Simulating a Bias Mitigation Scenario in Large Language Models
Department of Computer Science, Bowling Green State University(计算机科学系,布恩维尔州立大学);School of Computing, University of Nebraska–Lincoln(计算学院,内布拉斯加-林肯大学);Department of Civil, Environmental, and Ocean Engineering, Stevens Institute of Technology(土木、环境与海洋工程系,史蒂文斯理工学院);Department of Computer Science, University of South Carolina(计算机科学系,南卡罗来纳大学);Artificial Intelligence Institute, University of South Carolina(人工智能研究院,南卡罗来纳大学)
MF-LLM: Simulating Population Decision Dynamics via a Mean-Field Large Language Model Framework
Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所);School of Artificial Intelligence, Chinese Academy of Sciences(中国科学院大学人工智能学院);Nanyang Technological University(南洋理工大学);University of Bristol(布里斯托大学);Tianjin University(天津大学);Shanghai Jiao Tong University(上海交通大学);Renmin University of China(中国人民大学);University College London(伦敦大学学院)
AgentCAT: Simulating Computerized Adaptive Testing via Multi-Agent Large Language Models
Institute for Clarity in Documentation(文档清晰度研究所);Inria Paris-Rocquencourt(法国国家信息与自动化研究所巴黎-罗康库尔中心);Rajiv Gandhi University(拉吉夫·甘地大学);Tsinghua University(清华大学);Palmer Research Laboratories(帕默研究实验室);State Key Laboratory of Opto-Electronic Information Acquisition and Protection Technology, Institute of Physical Science and Information Technology, Anhui University(光电信息获取与防护技术国家重点实验室,物理科学与信息技术学院,安徽大学);School of Artificial Intelligence, Anhui University(安徽大学人工智能学院);School of Computer Science and Technology, Dalian University of Technology(大连理工大学计算机科学与技术学院);School of Computer Science and Technology, Anhui University(安徽大学计算机科学与技术学院);State Key Laboratory of Cognitive Intelligence, University of Science and Technology of China, the Institute of Artificial Intelligence, Hefei Comprehensive National Science Center(认知智能国家重点实验室,中国科学技术大学,人工智能研究院,合肥综合性国家科学中心)
CoderAgent: Simulating Student Behavior for Personalized Programming Learning with Large Language Models
State Key Laboratory of Cognitive Intelligence, University of Science and Technology of China(认知智能国家重点实验室,中国科学技术大学);Institute of Artificial Intelligence, Hefei Comprehensive National Science Center(人工智能研究院,合肥综合性国家科学中心)
Effectiveness of Large Language Models in Simulating Regional Psychological Structures: An Empirical Examination of Personality and Subjective Well-being
Tsinghua University(清华大学);Neoma Business School(诺欧商学院);Beijing Technology and Business University(北京工商大学);Beijing Normal University(北京师范大学)
Simulating multiple human perspectives in socio-ecological systems using large language models
Karlsruhe Institute of Technology(卡尔斯鲁厄理工学院);Highlands Rewilding Limited(高地野化有限公司);Bayes Business School(贝叶斯商学院);University of London(伦敦大学);University of Edinburgh(爱丁堡大学)
The Psychogenic Machine: Simulating AI Psychosis, Delusion Reinforcement and Harm Enablement in Large Language Models
King’s College Hospital(国王学院医院);Nuraxi AI Dev and Doc: AI for Healthcare(Nuraxi AI 人工智能医疗);Sage Bionetworks(Sage 生物网络);University College London(伦敦大学学院);King’s College London(国王学院伦敦)
Law in Silico: Simulating Legal Society with LLM-Based Agents
Institute for Artificial Intelligence, Peking University(北京大学人工智能研究所);Yuanpei College, Peking University(北京大学元培学院);School of Computing and Data Science, The University of Hong Kong(香港大学计算与数据科学学院);Law School, Peking University(北京大学法学院);State Key Laboratory of General Artificial Intelligence, BIGAI(通用人工智能国家重点实验室)
针对现实法律实验成本高、难开展的问题,提出基于LLM的法律社会模拟框架Law in Silico,可复现宏观犯罪趋势,还揭示了完善的法律系统对弱势群体的保护作用。