arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2606.09013cs.CL
收藏

超越平均值:在分布层面评估LLM对人类调查的复现能力

Beyond Averages: Evaluating LLMs on Human Survey Replication at the Distributional Level

Ewha Womans University(梨花女子大学)

Jeonghyeon Moon, Jiwon Kim, Yeheum Lah, Yoonju Han, Yuncheol Kang

AI总结:

本研究通过非公开的韩国方便面购买实验,在分布层面评估LLM复现人类调查响应的能力,发现均值匹配的模型可能产生更偏离人类的分布,且结构化角色和多模态输入提升对齐度,而推理提示则降低。

摘要:

LLM越来越多地被用于模拟人类调查响应,但先前的工作主要使用均值层面或总体一致性来评估复现能力,对LLM是否复现人类行为的变异性提供的见解有限。我们使用一个非公开的2010年韩国方便面购买消费者选择实验,在分布层面评估基于LLM的调查复现,该设置不太可能与模型训练数据重叠。我们评估了三种不同统计类型的响应变量:二元购买发生、分类品牌选择和计数购买数量。对于每种变量,我们在均值层面、模式和分布一致性上比较人类和LLM响应,并参考仅来自人类数据的基线。LLM在复现条件层面模式上表现合理,但未能捕捉分布结构:对于购买数量,没有模型能击败一个简单的条件不敏感基线(该基线仅匹配合并的人类分布)。因为均值匹配人类良好的模型仍可能产生比该基线更远离人类的分布,仅基于均值的评估可能具有误导性。复现能力也随输入配置而变化,结构化角色和多模态输入改善一致性,而显式推理提示则单调地降低一致性。

英文摘要:

LLMs are increasingly used to simulate human survey responses, but prior work has mainly evaluated replication using mean-level or aggregate agreement, offering limited insight into whether LLMs reproduce the variability of human behavior. We evaluate LLM-based survey replication at the distributional level using a non-public 2010 consumer choice experiment on Korean instant noodle purchases, a setting unlikely to overlap with model training data. We evaluate three response variables of differing statistical type: binary purchase incidence, categorical brand choice, and count purchase quantity. For each, we compare human and LLM responses at mean-level, pattern, and distributional alignment, and against reference baselines from the human data alone. LLMs reproduce condition-level patterns reasonably well but fail to capture distributional structure: for purchase quantity, no model beats a condition-insensitive baseline that simply matches the pooled human distribution. Because models that match human means well can still produce distributions further from humans than this baseline, mean-based evaluation alone can be actively misleading. Replication also varies with input configuration, with structured personas and multimodal inputs improving alignment while explicit reasoning prompting degrades it monotonically.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑