arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2607.03091cs.AIcs.CLcs.CYcs.MAstat.ME
收藏

通过跨调查转移进行硅采样

Silicon Sampling via Cross-Survey Transfer

National Sun Yat-sen University(国立中山大学);National Chung Hsing University(中国台湾国立中兴大学)

Chan-Tung Ku, Chan Hsu, Pei-Cing Huang, Frank Cheng-shan Liu, I-Ling Cheng, Yihuang Kang

AI总结:

研究提出跨调查转移评估框架,用大语言模型根据部分问题答案预测同调查中其他不同问题答案。利用2024年台湾选举与民主化研究数据等,发现零样本大语言模型精度及不同结构可预测性层级等,明确硅采样前景与局限。

摘要:

硅采样——利用大语言模型模拟人类调查受访者——已成为增强传统调查研究的一种有前景的方法。然而,大多数评估依赖分布比较而非个体层面预测,这有将模式匹配与连贯的受访者层面预测混淆的风险。我们提出跨调查转移,这是一个更严格的评估框架,其中大语言模型被给予受访者对一组问题的答案,并必须预测他们对来自同一调查的完全不同问题的答案。使用来自2024年台湾选举与民主化研究(TEDS)的数据、三个开放权重的大语言模型(27B - 120B参数)和监督机器学习基线,我们发现:(1)零样本大语言模型在真正未见过的项目上达到52%的准确率,接近在同群体数据上训练的监督随机森林的6个百分点(pp)以内;(2)出现了一个稳定的结构可预测性层级,从党派态度的67%到主权的23%;(3)方差崩溃和安全对齐效应——两个经常被引用的大语言模型限制——结果比以前报道的更细微,方差崩溃也影响监督模型,并且对齐效应在不同模型家族中差异很大。这些发现阐明了硅采样的前景和界限。

英文摘要:

Silicon sampling-using large language models (LLMs) to simulate human survey respondents-has emerged as a promising approach for augmenting traditional survey research. However, most evaluations rely on distributional comparisons rather than individual-level prediction, which risks conflating pattern matching with coherent respondent-level prediction. We propose cross-survey transfer, a more rigorous evaluation framework in which an LLM is given a respondent's answers to one set of questions and must predict their answers to entirely different questions from the same survey. Using data from the Taiwan Election and Democratization Study (TEDS) 2024, three open-weight LLMs (27B-120B parameters), and supervised machine learning baselines, we find that: (1) zero-shot LLMs achieve 52% accuracy on genuinely unseen items, closing to within 6 percentage points (pp) of a supervised random forest trained on same-population data; (2) a stable construct predictability hierarchy emerges, from 67% for partisan attitudes to 23% for sovereignty; and (3) variance collapse and safety alignment effects-two commonly cited LLM limitations-turn out to be more nuanced than previously reported, with variance collapse affecting supervised models as well and alignment effects varying dramatically across model families. These findings clarify both the promise and boundaries of silicon sampling.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑