使用大型语言模型模拟多个人类并复现人类受试者研究
Using Large Language Models to Simulate Multiple Humans and Replicate Human Subject Studies
AI总结：
本文提出图灵实验（TE）框架，用语言模型模拟人类受试者样本以复现经典行为实验，发现GPT-4等模型存在“超准确度偏差”，可能影响下游应用。
摘要：
我们引入一种新型测试，称为图灵实验（Turing Experiment, TE），用于评估给定的语言模型（如GPT模型）能在多大程度上模拟人类行为的不同方面。TE还可以揭示语言模型在模拟特定人类行为时存在的一致偏差。与涉及模拟单个任意个体的图灵测试不同，TE要求模拟人类受试者研究中具有代表性的参与者样本。我们开展了多个TE，尝试复现先前研究中已确立的发现。我们设计了一种模拟TE的方法论，并展示其用途，以比较不同语言模型在复现经典经济学、心理语言学和社交心理学实验方面的能力：最后通牒博弈、花园路径句、米尔格拉姆电击实验和群体智慧。在前三个TE中，使用近期模型复现了已有发现，而最后一个TE揭示了某些语言模型（包括ChatGPT和GPT-4）中存在的“超准确度偏差”，这可能影响教育和艺术领域的下游应用。
英文摘要：
We introduce a new type of test, called a Turing Experiment (TE), for evaluating to what extent a given language model, such as GPT models, can simulate different aspects of human behavior. A TE can also reveal consistent distortions in a language model's simulation of a specific human behavior. Unlike the Turing Test, which involves simulating a single arbitrary individual, a TE requires simulating a representative sample of participants in human subject research. We carry out TEs that attempt to replicate well-established findings from prior studies. We design a methodology for simulating TEs and illustrate its use to compare how well different language models are able to reproduce classic economic, psycholinguistic, and social psychology experiments: Ultimatum Game, Garden Path Sentences, Milgram Shock Experiment, and Wisdom of Crowds. In the first three TEs, the existing findings were replicated using recent models, while the last TE reveals a "hyper-accuracy distortion" present in some language models (including ChatGPT and GPT-4), which could affect downstream applications in education and the arts.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
