arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2305.09620cs.CLcs.AIcs.LG
收藏

AI增强的调查:利用大型语言模型和调查进行意见预测

AI-Augmented Surveys: Leveraging Large Language Models and Surveys for Opinion Prediction

Department of Sociology(社会学系);University of Chicago(芝加哥大学);New York University(纽约大学);Chicago, IL(伊利诺伊州芝加哥市);New York, NY(纽约州纽约市)

Junsol Kim, Byungkyu Lee

AI总结:

本文提出了一种基于大型语言模型的框架,通过结合问题、受访者和调查时期的嵌入表示,预测重复横断面调查中缺失的响应,从而弥补传统调查在捕捉历史变化方面的不足。

摘要:

全国代表性调查追踪公众意见,但每年只询问有限的问题,限制了其捕捉历史变化的潜力。为填补这一空白,我们开发了一个基于大型语言模型(LLM)的框架,通过结合问题、受访者和调查时期的嵌入表示,预测重复横断面调查中缺失的响应。我们引入了LLM在调查研究中的两个新应用:回溯预测(预测年度层面的缺失意见)和未询问意见预测(预测完全缺失的意见)。使用1972-2021年一般社会调查的数据,我们的LLM模型在交叉验证和在GSS未询问的年份中通过其他组织测量的公众意见方面表现良好。这些能力使我们能够恢复缺失的趋势并确定公众态度变化的时间,例如同性婚姻支持率的上升。然而,未询问意见预测的性能仍较为有限。我们展示了当我们的模型优于现有基准时的情况,检验了哪些意见和受访者更具可预测性,并评估了我们的方法是否减少了LLM预测响应的同质化倾向。我们的研究证明了LLM和调查可以相互增强:LLM扩大了调查的潜力,而调查则校准LLM以模拟人类意见。

英文摘要:

Nationally representative surveys track public opinion, yet they ask only a limited set of questions each year, limiting its potential to capture historical changes. To fill this gap, we develop a large language model (LLM)-based framework for predicting missing responses in repeated cross-sectional surveys by incorporating embeddings for questions, respondents, and survey periods. We introduce two new applications of LLMs to survey research: retrodiction (predicting year-level missing opinions) and unasked opinion prediction (predicting entirely missing opinions). Using data from the 1972-2021 General Social Surveys, our LLM-based models perform strongly in retrodicting masked GSS opinions through cross-validation and public opinions measured by other organizations in years when the GSS did not ask them. These capabilities enable us to recover missing trends and pinpoint when public attitudes changed, such as the rising support for same-sex marriage. However, performance remains modest for unasked opinion prediction. We show when our models outperform established benchmarks, examine which opinions and and respondents are more predictable, and evaluate whether our approach reduces LLMs' tendency to homogenize predicted responses. Our study demonstrates that LLMs and surveys can mutually enhance each other: LLMs broaden survey potential, while surveys calibrate LLMs for simulating human opinions.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑