arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2602.18462cs.CYcs.AI
收藏

评估基于人设的大型语言模型作为合成调查受访者可靠性

Assessing the Reliability of Persona-Conditioned LLMs as Synthetic Survey Respondents

IIT-CNR, University of Pisa(IIT-CNR与比萨大学);University of Florence(佛罗伦萨大学);University of Pisa(比萨大学)

Erika Elizabeth Taday Morocho, Lorenzo Cima, Tiziano Fagni, Marco Avvenuti, Stefano Cresci

AI总结:

本文评估了基于人设的LLM作为合成调查受访者可靠性,发现多属性提示可能引入偏差,影响子群体的忠实性。

摘要:

在计算社会科学和基于代理的模拟中,使用基于人设的大型语言模型(LLM)作为合成调查受访者已成为一种常见实践。然而,多属性人设提示是否能提高LLM的可靠性,还是反而引入了偏差,仍不清楚。本文通过利用来自世界价值观调查的美国微观数据集,对这一问题进行了评估。具体而言,我们对两个开源权重聊天模型和一个随机猜测基线模型,在超过7万次受访者-项目实例上进行了评估。我们发现,人设提示并未带来调查一致性方面的明显整体提升,且在许多情况下显著降低了性能。人设效应高度异质,大多数项目表现出最小变化,而一小部分问题和少数族裔子群体则经历了不成比例的偏差。我们的发现突显了当前基于人设的模拟实践的一个关键负面影响:人口统计学条件可能以削弱子群体忠实性和风险误导下游分析的方式重新分配误差。

英文摘要:

Using persona-conditioned LLMs as synthetic survey respondents has become a common practice in computational social science and agent-based simulations. Yet, it remains unclear whether multi-attribute persona prompting improves LLM reliability or instead introduces distortions. Here we contribute to this assessment by leveraging a large dataset of U.S. microdata from the World Values Survey. Concretely, we evaluate two open-weight chat models and a random-guesser baseline across more than 70K respondent-item instances. We find that persona prompting does not yield a clear aggregate improvement in survey alignment and, in many cases, significantly degrades performance. Persona effects are highly heterogeneous as most items exhibit minimal change, while a small subset of questions and underrepresented subgroups experience disproportionate distortions. Our findings highlight a key adverse impact of current persona-based simulation practices: demographic conditioning can redistribute error in ways that undermine subgroup fidelity and risk misleading downstream analyses.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑