大型语言模型是变色龙吗？一次模拟社会调查的尝试
Are Large Language Models Chameleons? An Attempt to Simulate Social Surveys
AI总结：
本文通过数百万次模拟探究LLMs能否模拟社会调查，发现提示会引发显著文化、年龄和性别偏差，并提出基于Jaccard相似度的新度量以比较LLM回答与ESS数据的差异，强调使用前须分析提示的鲁棒性与变异性。
摘要：
大型语言模型（LLMs）能否模拟社会调查？为回答这一问题，我们进行了数百万次模拟，要求LLMs回答主观性问题。将不同LLM的回答与欧洲社会调查（ESS）数据进行比较后发现，提示对偏差和变异性的影响是根本性的，凸显了显著的文化、年龄和性别偏差。我们进一步讨论了用于衡量LLM回答与调查数据之间差异的统计方法，并提出了一种受Jaccard相似度启发的新度量，因为LLM生成的回答可能具有较小的方差。我们的实验还揭示，在使用LLMs模拟社会调查之前，分析提示的鲁棒性和变异性非常重要，因为它们的模仿能力充其量只是近似的。
英文摘要：
Can large language models (LLMs) simulate social surveys? To answer this question, we conducted millions of simulations in which LLMs were asked to answer subjective questions. A comparison of different LLM responses with the European Social Survey (ESS) data suggests that the effect of prompts on bias and variability is fundamental, highlighting major cultural, age, and gender biases. We further discussed statistical methods for measuring the difference between LLM answers and survey data and proposed a novel measure inspired by Jaccard similarity, as LLM-generated responses are likely to have a smaller variance. Our experiments also reveal that it is important to analyze the robustness and variability of prompts before using LLMs to simulate social surveys, as their imitation abilities are approximate at best.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
