LLMs are increasingly used to simulate human survey responses, but prior work has mainly evaluated replication using mean-level or aggregate agreement, offering limited insight into whether LLMs reproduce the variability of human behavior. We evaluate LLM-based survey replication at the distributional level using a non-public 2010 consumer choice experiment on Korean instant noodle purchases, a setting unlikely to overlap with model training data. We evaluate three response variables of differing statistical type: binary purchase incidence, categorical brand choice, and count purchase quantity. For each, we compare human and LLM responses at mean-level, pattern, and distributional alignment, and against reference baselines from the human data alone. LLMs reproduce condition-level patterns reasonably well but fail to capture distributional structure: for purchase quantity, no model beats a condition-insensitive baseline that simply matches the pooled human distribution. Because models that match human means well can still produce distributions further from humans than this baseline, mean-based evaluation alone can be actively misleading. Replication also varies with input configuration, with structured personas and multimodal inputs improving alignment while explicit reasoning prompting degrades it monotonically.
ReplicatorBench: Benchmarking LLM Agents for Replicability in Social and Behavioral Sciences
University of Notre Dame(圣母大学);Old Dominion University(老道明大学);Pennsylvania State University(宾夕法尼亚州立大学);Independent Researcher(独立研究员);Uppsala University(乌普萨拉大学);Center for Open Science(开放科学中心)
K-Bench: a clinically calibrated benchmark for evaluating large language models in high-risk mental health conversations
University of Roehampton(罗汉普顿大学);Kivira Health;University of Hertfordshire(赫特福德大学);University of Surrey(萨里大学);University of Bedfordshire(贝德福德大学);Tavistock Relationships(塔维斯托克关系中心);InsideOut
From Demographics to Survey Anchors: Evaluating LLM Agents for Modeling Retirement Attitudes
AI Research(AI研究);AXA Group Operations(AXA集团运营);CDSP & CEE(CDSP与CEE);Sciences Po(社会科学高等学院);Computer Science Department(计算机科学系);Stanford University(斯坦福大学)
Distributional Metrics for Evaluating Spoken Conversational Systems
KTH Royal Institute of Technology(KTH皇家理工学院);NICT(日本信息通信研究机构);Charles University(查理大学);Karlsruhe Institute of Technology(卡尔斯鲁厄理工学院);University of Edinburgh(爱丁堡大学)