评估LLMs在受控实验中作为人类替代品的效用
Evaluating LLMs as Human Surrogates in Controlled Experiments
AI总结:
本文通过比较LLM生成数据与人类数据,探讨在准确性感知实验中LLM能否作为人类替代品,发现LLM能复现部分人类效应,但效果大小和调节模式因模型而异。
摘要:
大型语言模型(LLMs)越来越多地用于模拟人类反应在行为研究中,但尚不清楚何时LLM生成的数据能支持与人类数据相同的实验推断。我们通过直接比较现成的LLM生成响应与人类响应,对这一问题进行了评估。在经典的准确性感知调查实验中,每条人类观察都被转换为结构化提示,模型生成一个0-10的输出变量,且不进行任务特定训练;对人类和合成响应应用相同的统计分析。我们发现,LLMs复现了人类观察到的几个方向效应,但效果大小和调节模式在不同模型中有所不同。因此,现成的LLMs在受控条件下能够捕捉到群体信念更新模式,但不能一致地匹配人类规模的效果,明确了LLM生成的数据何时可以作为行为替代品。
英文摘要:
Large language models (LLMs) are increasingly used to simulate human responses in behavioral research, yet it remains unclear when LLM-generated data support the same experimental inferences as human data. We evaluate this by directly comparing off-the-shelf LLM-generated responses with human responses from a canonical survey experiment on accuracy perception. Each human observation is converted into a structured prompt, and models generate a single 0--10 outcome variable without task-specific training; identical statistical analyses are applied to human and synthetic responses. We find that LLMs reproduce several directional effects observed in humans, but effect magnitudes and moderation patterns vary across models. Off-the-shelf LLMs therefore capture aggregate belief-updating patterns under controlled conditions but do not consistently match human-scale effects, clarifying when LLM-generated data can function as behavioral surrogates.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。