arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2405.07248cs.CLcs.AIcs.CYcs.HC
收藏

大语言模型模拟人类心理行为的能力有限:一项心理测量学分析

Limited Ability of LLMs to Simulate Human Psychological Behaviours: a Psychometric Analysis

University of Cambridge(剑桥大学)

Nikolay B Petrov, Gregory Serapio-García, Jason Rentfrow

AI总结:

本研究采用心理测量学方法,测试GPT-3.5和GPT-4在不同人设下的人格问卷回答,发现大语言模型模拟人类心理行为的能力有限,尤其在特定人口统计人设下心理测量学属性较差。

摘要:

大语言模型(LLMs)类人化的回答促使社会科学家研究其是否可用于模拟实验、民意测验和调查中的人类参与者。这类研究的核心关注点是通过让大语言模型回答标准化问卷来描绘其心理特征。由于从大语言模型对问卷的文本回答中推断潜在特质并非易事,这类研究得出相互矛盾的结果也就不足为奇了。为解决这一问题,我们采用了心理测量学这一心理测量科学。在本研究中,我们让OpenAI的旗舰模型GPT-3.5和GPT-4扮演不同的人设,并回答一系列人格特质的标准化测量量表。我们使用了两种人设描述:通用型(4到5个随机人物描述)和特定型(主要来自大规模人类数据集的真实人口统计特征)。我们发现,使用通用人设描述时,GPT-4(而非GPT-3.5)的回答展现出虽不完美但颇具前景的心理测量学属性,与人类常模相似;但当使用特定人口统计特征时,两个大语言模型的数据都表现出较差的心理测量学属性。我们得出结论:目前当要求大语言模型模拟“硅基人设”时,其回答无法有效反映潜在的心理特质。因此,我们的研究对大语言模型在多选题回答任务中模拟个体层面人类行为的能力提出了质疑。

英文摘要:

The humanlike responses of large language models (LLMs) have prompted social scientists to investigate whether LLMs can be used to simulate human participants in experiments, opinion polls and surveys. Of central interest in this line of research has been mapping out the psychological profiles of LLMs by prompting them to respond to standardized questionnaires. The conflicting findings of this research are unsurprising given that mapping out underlying, or latent, traits from LLMs' text responses to questionnaires is no easy task. To address this, we use psychometrics, the science of psychological measurement. In this study, we prompt OpenAI's flagship models, GPT-3.5 and GPT-4, to assume different personas and respond to a range of standardized measures of personality constructs. We used two kinds of persona descriptions: either generic (four or five random person descriptions) or specific (mostly demographics of actual humans from a large-scale human dataset). We found that the responses from GPT-4, but not GPT-3.5, using generic persona descriptions show promising, albeit not perfect, psychometric properties, similar to human norms, but the data from both LLMs when using specific demographic profiles, show poor psychometrics properties. We conclude that, currently, when LLMs are asked to simulate silicon personas, their responses are poor signals of potentially underlying latent traits. Thus, our work casts doubt on LLMs' ability to simulate individual-level human behaviour across multiple-choice question answering tasks.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑