大型语言模型在模拟调查用户响应中的分析
An Analysis of Large Language Models for Simulating User Responses in Surveys
AI总结：
本文分析了大型语言模型在模拟调查用户响应中的表现，提出CLAIMSIM方法以提高响应多样性，但发现模型在处理不同人口特征时存在固有偏见和推理限制。
摘要：
利用大型语言模型（LLMs）来模拟用户意见已受到越来越多的关注。然而，尤其是经过强化学习从人类反馈（RLHF）训练的LLMs，已知会倾向于主流观点，这引发了对其能否代表来自不同人口和文化背景的用户能力的担忧。在本工作中，我们通过直接提示和思考链提示来考察LLMs在模拟跨领域调查问题的人类响应方面的程度。我们进一步提出了一种主张多样化方法CLAIMSIM，该方法将LLM参数知识作为上下文输入来获取观点。在调查问题回答任务上的实验表明，尽管CLAIMSIM产生了更多样化的响应，但两种方法在准确模拟用户方面都遇到困难。进一步分析揭示了两个关键限制：（1）LLMs在不同人口特征下倾向于保持固定观点，并生成单一视角的主张；（2）当面对冲突的主张时，LLMs在处理人口特征之间的细微差异时遇到困难，限制了其适应特定用户档案的能力。
英文摘要：
Using Large Language Models (LLMs) to simulate user opinions has received growing attention. Yet LLMs, especially trained with reinforcement learning from human feedback (RLHF), are known to exhibit biases toward dominant viewpoints, raising concerns about their ability to represent users from diverse demographic and cultural backgrounds. In this work, we examine the extent to which LLMs can simulate human responses to cross-domain survey questions through direct prompting and chain-of-thought prompting. We further propose a claim diversification method CLAIMSIM, which elicits viewpoints from LLM parametric knowledge as contextual input. Experiments on the survey question answering task indicate that, while CLAIMSIM produces more diverse responses, both approaches struggle to accurately simulate users. Further analysis reveals two key limitations: (1) LLMs tend to maintain fixed viewpoints across varying demographic features, and generate single-perspective claims; and (2) when presented with conflicting claims, LLMs struggle to reason over nuanced differences among demographic features, limiting their ability to adapt responses to specific user profiles.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
