The humanlike responses of large language models (LLMs) have prompted social scientists to investigate whether LLMs can be used to simulate human participants in experiments, opinion polls and surveys. Of central interest in this line of research has been mapping out the psychological profiles of LLMs by prompting them to respond to standardized questionnaires. The conflicting findings of this research are unsurprising given that mapping out underlying, or latent, traits from LLMs' text responses to questionnaires is no easy task. To address this, we use psychometrics, the science of psychological measurement. In this study, we prompt OpenAI's flagship models, GPT-3.5 and GPT-4, to assume different personas and respond to a range of standardized measures of personality constructs. We used two kinds of persona descriptions: either generic (four or five random person descriptions) or specific (mostly demographics of actual humans from a large-scale human dataset). We found that the responses from GPT-4, but not GPT-3.5, using generic persona descriptions show promising, albeit not perfect, psychometric properties, similar to human norms, but the data from both LLMs when using specific demographic profiles, show poor psychometrics properties. We conclude that, currently, when LLMs are asked to simulate silicon personas, their responses are poor signals of potentially underlying latent traits. Thus, our work casts doubt on LLMs' ability to simulate individual-level human behaviour across multiple-choice question answering tasks.
Are LLMs effective psychological assessors? Leveraging adaptive RAG for interpretable mental health screening through psychometric practice
Euler Institute, Università della Svizzera italiana(欧拉研究所,瑞士大学);National Institute of Informatics(国家信息研究所);Brown University(布朗大学);University of Tübingen(图宾根大学);Insubria University(因斯布里亚大学)
PsychoGAT: A Novel Psychological Measurement Paradigm through Interactive Fiction Games with LLM Agents
Tsinghua University(清华大学);The Second Xiangya Hospital, Central South University(中南大学湘雅二医院);Medical Psychological Institute, Central South University(中南大学医学心理学研究所);National Clinical Research Center for Mental Disorders(国家精神疾病临床研究中心);Carnegie Mellon University(卡内基梅隆大学)
Large Language Model Psychometrics: A Systematic Review of Evaluation, Validation, and Enhancement
State Key Laboratory of General Artificial Intelligence, School of Intelligence Science and Technology, Peking University(一般人工智能国家重点实验室,智能科学与技术学院,北京大学);School of Psychological and Cognitive Sciences, Peking University(心理学与认知科学学院,北京大学);Key Laboratory of Machine Perception (Ministry of Education), Peking University(机器感知重点实验室(教育部),北京大学)
AIPsychoBench: Understanding the Psychometric Differences between LLMs and Humans
College of Computer Science and Technology, National University of Defense Technology(国防科技大学计算机科学与技术学院);Institute of High Performance Computing, Agency for Science, Technology and Research (A*STAR)(科学研究院高性能计算研究所);Institute of Information Engineering, Chinese Academy of Sciences(中国科学院信息工程研究所)
Microsoft Research Asia(微软亚洲研究院);Beijing Normal University(北京师范大学);Universitat Politècnica de València(瓦伦西亚理工大学);ValgrAI;The Psychometrics Centre, University of Cambridge(剑桥大学心理测量中心);Cambridge Judge Business School(剑桥大学嘉治商学院)
The Ultimate Tutorial for AI-driven Scale Development in Generative Psychometrics: Releasing AIGENIE from its Bottle
Department of Psychology, University of Virginia(弗吉尼亚大学心理学系);Department of Psychology, Pontificia Universidad Madre y Maestra(宗座圣母大学心理学系);Department of Psychology, Vanderbilt University(范德比尔特大学心理学系)
Measuring Human and AI Values Based on Generative Psychometrics with Large Language Models
State Key Laboratory of General Artificial Intelligence(通用人工智能国家重点实验室);School of Intelligence Science and Technology, Peking University(北京大学智能科学技术学院);School of Software and Microelectronics, Peking University(北京大学软件与微电子学院);School of Psychological and Cognitive Sciences, Peking University(北京大学心理与认知科学学院);PKU-Wuhan Institute for Artificial Intelligence(北京大学武汉人工智能研究院)
GenPT: Beyond Self-Report for Reliable LLM Psychometrics via Generative Projective Testing
School of Computer Science and Engineering, Northeastern University(东北大学计算机科学与工程学院);School of Computing and Information Systems, Singapore Management University(新加坡管理学院计算机与信息学院);Mental Health Education Center, Northeastern University(东北大学心理健康教育中心);School of Psychology, Northeast Normal University(东北师范大学心理学系);Faculty of psychology, Southwest University(西南大学心理学系);School of Sociology and Psychology, Central University of Finance and Economics(中央财经大学社会学与心理学学院);College of Arts, Northeastern University(东北大学艺术学院)
Measuring Behavioural Signatures of Large Language Models through Psychometric Profiling
The Chinese University of Hong Kong, Shenzhen(香港中文大学(深圳));Central China Normal University(华中师范大学);Shandong Science and Technology Press, Shandong Publishing Group(山东科学技术出版社,山东出版集团);Bielefeld University(比勒费尔德大学);South China Normal University(华南师范大学);Frankfurt Institute for Advanced Studies(法兰克福高等研究院)
Human Psychometric Questionnaires Mischaracterize LLM Behavior
Graduate School of Data Science, Seoul National University(首尔国立大学数据科学研究生院);Department of Communication, Interdisciplinary Program in Artificial Intelligence, Seoul National University(首尔国立大学通信系人工智能交叉学科项目)
Psychometric Item Validation Using Virtual Respondents with Trait-Response Mediators
Graduate School of Data Science, Seoul National University(首尔国立大学数据科学研究生院);Department of Communication, Seoul National University(首尔国立大学通信系);Interdisciplinary Program in Artificial Intelligence, Seoul National University(首尔国立大学人工智能跨学科项目)
Generative Personality Simulation via Theory-Informed Structured Interview
Department of Psychological Sciences & 2 Department of Computer Science, Rice University(1 心理学系 & 2 计算机科学系,莱斯大学);Department of Electrical and Computer Engineering, Northeastern University(3 电气与计算机工程系,东北大学);Department of Computer Science, Johns Hopkins University(4 计算机科学系,约翰霍普金斯大学);Microsoft Research Asia(微软亚洲研究院)
Who We Are, Where We Are: Mental Health at the Intersection of Person, Situation, and Large Language Models
Department of Computer Science, Stony Brook University(石英布大学计算机科学系);Oslo Metropolitan University, Oslo Business School(奥斯陆 Metropolitan 大学奥斯陆商学院);University of Texas at Dallas, Dept. of Psychology(德克萨斯大学达拉斯分校心理学系)
Apparent Psychological Profiles of Large Language Models are Largely a Measurement Artifact
Max Planck Institute for Human Development(马克斯·普朗克人类发展研究所);University of Konstanz(康斯坦茨大学);Barcelona Supercomputing Center(巴塞罗那超级计算中心);University of Basel(巴塞尔大学)
Enhancing Depression-Diagnosis-Oriented Chat with Psychological State Tracking
School of Computer Science and Technology, East China Normal University(东华大学计算机科学与技术学院);Shanghai Institute of Al for Education, East China Normal University(东华大学上海人工智能教育研究所);School of Psychology and Cognitive Science, East China Normal University(东华大学心理学与认知科学学院)
Effectiveness of Large Language Models in Simulating Regional Psychological Structures: An Empirical Examination of Personality and Subjective Well-being
Tsinghua University(清华大学);Neoma Business School(诺欧商学院);Beijing Technology and Business University(北京工商大学);Beijing Normal University(北京师范大学)
KokoroChat: A Japanese Psychological Counseling Dialogue Dataset Collected via Role-Playing by Trained Counselors
The University of Electro-Communications(电通通讯大学);Rapport Technologies, Inc.(Rapport技术公司);iDEAR Human Support Service(iDEAR人类支持服务);Japanese Organization of Mental Health and Educational Agencies(日本心理健康与教育机构联合会)
Scaling Law in LLM Simulated Personality: More Detailed and Realistic Persona Profile Is All You Need
Department of Computer;Information Engineering, Hebei Petroleum University of Technology, Chengde, China (e-mail;School of Computer Science \& Technology, Beijing Institute of Technology, Beijing, China (e-mail
"Dark Triad" Model Organisms of Misalignment: Narrow Fine-Tuning Mirrors Human Antisocial Behavior
Brain & Creativity Institute, University of Southern California(大脑与创造力研究所,南加州大学);Department of Psychology, University of Southern California(心理学系,南加州大学);Interchange Forum for Reflecting on Intelligent Systems, University of Stuttgart(智能系统反思交流论坛,斯图加特大学)
Evaluating the Efficacy of Interactive Language Therapy Based on LLM for High-Functioning Autistic Adolescent Psychological Counseling
Branksome Hall Asia(亚洲布兰克森霍尔学院);Algorix Convergence Research Office(Algorix融合研究室);Korea Science Academy of KAIST(韩国科学技术院附属韩国科学英才学校);Hana Academy Seoul(首尔汉拿学院)
Position: Stop Evaluating AI with Human Tests, Develop Principled, AI-specific Tests instead
Max Planck Institute for Intelligent Systems, Tübingen(马克斯·普朗克智能系统研究所,图宾根);ETH Zurich, Zurich(苏黎世联邦理工学院,苏黎世);University of Tübingen, Methods Center, Tübingen(图宾根大学,方法中心,图宾根);Massachusetts Institute of Technology, Cambridge(麻省理工学院,剑桥)