发表机构
GESIS – Leibniz Institute for the Social Sciences; University of Mannheim; Complexity Science Hub; RWTH Aachen University（GESIS——莱布尼茨社会科学研究所; 曼海姆大学; 复杂科学中心; 亚琛工业大学）
机构由 AI 辅助整理，请以论文原文为准。AI 中文总结
本文针对角色提示在LLM调查预测中效果混合的问题，提出人类反应差异是潜在解释，对比不同角色属性选择方法，在多国多调查多模型任务上评估，明确角色提示适用场景并提供属性选择方法新见解。
AI 中文摘要
大语言模型（LLMs）越来越多地被用于预测调查小组中人类参与者的反应。为实现这一目标，角色提示（persona prompting）近年来成为一种用于告知和对齐大型预训练语言模型的技术。角色提示指在提示中使用“角色”的简短文本描述来引导LLMs生成内容的实践。角色通过不同属性（如社会人口统计学特征、态度或行为）描述个体，旨在对齐LLMs以生成与对应人类反应相关的响应。然而，近期研究得出了关于角色提示的混合且部分相互矛盾的结果，缺乏明确的成功与失败模式。少数一致的发现包括：角色属性的选择至关重要，且使用更多属性未必能带来更好的性能。目前仍不清楚不同的属性选择方法表现如何，以及应如何在它们之间进行选择。在本文中，我们提出，调查问题中观察到的人类反应差异是迄今为止观察到的混合性能的潜在解释。此外，我们比较了与不同角色属性选择方法相关的角色提示的性能。我们在两个国家的四项不同（通用）社会调查、六个LLMs以及每项调查的二十项预测任务上评估这些方法。我们的工作有助于确定角色提示何时有望在调查预测任务中发挥作用，并为使用角色提示的基于LLM的调查预测中不同属性选择方法的有效性提供新见解。
英文摘要
Large Language Models (LLMs) are increasingly used to predict the responses of human participants in survey panels. Towards that goal, persona prompting has recently emerged as a technique to inform and align large pretrained language models. Persona prompting refers to the practice of using short textual descriptions of 'personas' in prompts to steer the LLM's generations. Personas describe individuals through different attributes such as their socio-demographics, attitudes, or behaviors, with the aim of aligning LLMs to produce responses that correlate with the corresponding human responses. Yet, recent work has produced mixed and partly conflicting results of persona prompting without clear patterns of success and failure. Among the few consistent findings is that the selection of persona attributes matters, and that using more attributes does not necessarily lead to better performance. It remains unclear how different attribute selection methods perform and how to choose among them. In this paper, we propose that observed human response variation of a survey question is a potential explanation for the mixed performance observed so far. In addition, we compare the performance of persona prompting associated with different methods for selecting persona attributes. We evaluate these methods on four different (general) social surveys across two countries, six LLMs, and twenty prediction tasks per survey. Our work helps to identify when persona prompting can be expected to be useful in survey prediction tasks, and provides new insights on the effectiveness of different attribute selection methods for LLM-based survey prediction using persona prompting.
Comments45 pages, 15 figures
