Large Language Models (LLMs) have demonstrated exceptional capabilities in solving various tasks, progressively evolving into general-purpose assistants. The increasing integration of LLMs into society has sparked interest in whether they exhibit psychological patterns, and whether these patterns remain consistent across different contexts -- questions that could deepen the understanding of their behaviors. Inspired by psychometrics, this paper presents a {comprehensive benchmark for quantifying psychological constructs of LLMs}, encompassing psychological dimension identification, assessment dataset design, and assessment with results validation. Our work identifies five key psychological constructs -- personality, values, emotional intelligence, theory of mind, and self-efficacy -- assessed through a suite of 13 datasets featuring diverse scenarios and item types. We uncover significant discrepancies between LLMs' self-reported traits and their response patterns in real-world scenarios, revealing complexities in their behaviors. Our findings also show that some preference-based tests, originally designed for humans, could not solicit reliable responses from LLMs. This paper offers a thorough psychometric assessment of LLMs, providing insights into reliable evaluation and potential applications in AI and social sciences.
Evaluating LLM-as-a-Judge Beyond Score Alignment: A Psychometric Analysis of Residual Judging Difficulty
DIPF | Leibniz Institute for Research and Information in Education(DIPF | 莱布尼茨教育与研究信息研究所);Centre for International Student Assessment (ZIB)(国际学生评估中心(ZIB));Faculty of Computer Science, Goethe University Frankfurt(法兰克福大学计算机科学学院);Chemnitz University of Technology(开姆尼茨工业大学)
Defining and Evaluating Visual Language Models' Basic Spatial Abilities: A Perspective from Psychometrics
School of Architecture, Tsinghua University(清华大学建筑学院);Department of Electronic Engineering, Tsinghua University(清华大学电子工程系);BNRist, Tsinghua University(清华大学BNRist)
Evaluating Implicit Bias in Large Language Models by Attacking From a Psychometric Perspective
State Key Laboratory of AI Safety, Institute of Computing Technology, Chinese Academy of Sciences(中国科学院人工智能安全国家重点实验室,计算技术研究所);Key Laboratory of Network Data Science and Technology, Institute of Computing Technology, Chinese Academy of Sciences(中国科学院网络数据科学与技术重点实验室,计算技术研究所);University of Chinese Academy of Sciences(中国科学院大学)
ValueBench: Towards Comprehensively Evaluating Value Orientations and Understanding of Large Language Models
Peking University(北京大学);National Key Laboratory of General Artificial Intelligence(通用人工智能国家重点实验室);School of Intelligence Science and Technology, Peking University(北京大学智能科学与技术学院);Department of Sociology, Peking University(北京大学社会学系);School of Psychological and Cognitive Sciences, Peking University(北京大学心理与认知科学学院);PKU-Wuhan Institute for Artificial Intelligence(北京大学武汉人工智能研究院)
Measuring Human and AI Values Based on Generative Psychometrics with Large Language Models
State Key Laboratory of General Artificial Intelligence(通用人工智能国家重点实验室);School of Intelligence Science and Technology, Peking University(北京大学智能科学技术学院);School of Software and Microelectronics, Peking University(北京大学软件与微电子学院);School of Psychological and Cognitive Sciences, Peking University(北京大学心理与认知科学学院);PKU-Wuhan Institute for Artificial Intelligence(北京大学武汉人工智能研究院)
Large Language Model Psychometrics: A Systematic Review of Evaluation, Validation, and Enhancement
State Key Laboratory of General Artificial Intelligence, School of Intelligence Science and Technology, Peking University(一般人工智能国家重点实验室,智能科学与技术学院,北京大学);School of Psychological and Cognitive Sciences, Peking University(心理学与认知科学学院,北京大学);Key Laboratory of Machine Perception (Ministry of Education), Peking University(机器感知重点实验室(教育部),北京大学)
Measuring Behavioural Signatures of Large Language Models through Psychometric Profiling
The Chinese University of Hong Kong, Shenzhen(香港中文大学(深圳));Central China Normal University(华中师范大学);Shandong Science and Technology Press, Shandong Publishing Group(山东科学技术出版社,山东出版集团);Bielefeld University(比勒费尔德大学);South China Normal University(华南师范大学);Frankfurt Institute for Advanced Studies(法兰克福高等研究院)
PsyScore: A Psychometrically-Aware Framework for Trait-Adaptive Essay Scoring and ZPD-Scaffolded Feedback
Department of Educational Psychology, East China Normal University(华东师范大学教育心理学系);Shanghai Institute of Artificial Intelligence for Education, East China Normal University(华东师范大学上海智能教育研究院);School of Computer Science and Technology, East China Normal University(华东师范大学计算机科学与技术学院)
AIPsychoBench: Understanding the Psychometric Differences between LLMs and Humans
College of Computer Science and Technology, National University of Defense Technology(国防科技大学计算机科学与技术学院);Institute of High Performance Computing, Agency for Science, Technology and Research (A*STAR)(科学研究院高性能计算研究所);Institute of Information Engineering, Chinese Academy of Sciences(中国科学院信息工程研究所)
Neural network embeddings recover value dimensions from psychometric survey items on par with human data
Barcelona Supercomputing Center(巴塞罗那超级计算中心);Jam Technologies GmbH;University of Mannheim(曼海姆大学);GESIS - Leibniz Institute for the Social Sciences(莱布尼茨社会科学研究所)
Evaluating and Inducing Personality in Pre-trained Language Models
Institute for Artificial Intelligence, Peking University(北京大学人工智能研究院);Yuanpei College, Peking University(北京大学元培学院);National Key Laboratory of General Artificial Intelligence, BIGAI(北京通用人工智能研究院通用人工智能全国重点实验室);Beijing Jiaotong University(北京交通大学)
GenPT: Beyond Self-Report for Reliable LLM Psychometrics via Generative Projective Testing
School of Computer Science and Engineering, Northeastern University(东北大学计算机科学与工程学院);School of Computing and Information Systems, Singapore Management University(新加坡管理学院计算机与信息学院);Mental Health Education Center, Northeastern University(东北大学心理健康教育中心);School of Psychology, Northeast Normal University(东北师范大学心理学系);Faculty of psychology, Southwest University(西南大学心理学系);School of Sociology and Psychology, Central University of Finance and Economics(中央财经大学社会学与心理学学院);College of Arts, Northeastern University(东北大学艺术学院)
The Ultimate Tutorial for AI-driven Scale Development in Generative Psychometrics: Releasing AIGENIE from its Bottle
Department of Psychology, University of Virginia(弗吉尼亚大学心理学系);Department of Psychology, Pontificia Universidad Madre y Maestra(宗座圣母大学心理学系);Department of Psychology, Vanderbilt University(范德比尔特大学心理学系)
Psychometric Item Validation Using Virtual Respondents with Trait-Response Mediators
Graduate School of Data Science, Seoul National University(首尔国立大学数据科学研究生院);Department of Communication, Seoul National University(首尔国立大学通信系);Interdisciplinary Program in Artificial Intelligence, Seoul National University(首尔国立大学人工智能跨学科项目)
PATCH! {P}sychometrics-{A}ssis{T}ed Ben{CH}marking of Large Language Models against Human Populations: A Case Study of Proficiency in 8th Grade Mathematics
PersonaTwin: A Multi-Tier Prompt Conditioning Framework for Generating and Evaluating Personalized Digital Twins
Viterbi School of Engineering, University of Southern California(南加州大学维特比工程学院);Human-centered Analytics Lab, University of Notre Dame(诺特大学人本分析实验室);Department of IT, Analytics, and Operations, University of Notre Dame(诺特大学信息技术、分析与运营部门);Department of Information Systems, Business Statistics and Operations Management, HKUST(香港科技大学信息系统、商业统计与运营管理部门)
Undesirable Biases in NLP: Addressing Challenges of Measurement
Institute for Logic, Language and Computation, University of Amsterdam(阿姆斯特丹大学逻辑、语言与计算研究所);Department of Experimental Psychology, Utrecht University(乌得勒支大学实验心理学系)
Are LLMs effective psychological assessors? Leveraging adaptive RAG for interpretable mental health screening through psychometric practice
Euler Institute, Università della Svizzera italiana(欧拉研究所,瑞士大学);National Institute of Informatics(国家信息研究所);Brown University(布朗大学);University of Tübingen(图宾根大学);Insubria University(因斯布里亚大学)