Human psychometric questionnaires are increasingly used to characterize large language models (LLMs), but it remains unclear what exactly their responses reflect. To answer this question we analyze responses from 50 LLMs across 45 questionnaires spanning multiple psychological domains using psychometric methodology. We found a dominant principal component explaining 47.1% of variance across standardized, response-derived questionnaire-level primary scores, revealing substantial shared structure across instruments. We investigate its meaning through semantic analysis, respondent framing, and human annotations. Semantic analysis suggests that the component partly reflects experiential self-attribution: how readily models endorse statements describing subjective states and bodily experiences. A model score emphasizing items whose disagreement decreases under human simulation correlates strongly with the component ($r=0.864$). Human ratings of 289 matched items show that inner-life content is positively associated, and social desirability negatively associated, with item-component correlations after mutual adjustment and controls for other semantic properties. Together, six rated properties explain 22.1% of variance in Fisher-transformed item-component correlations, supporting a partial interpretation involving both experiential self-attribution and socially desirable responding. These findings suggest that LLM psychometric profiles share response tendencies that require interpretation beyond the human traits their questionnaires were designed to measure.
Evaluating LLM-as-a-Judge Beyond Score Alignment: A Psychometric Analysis of Residual Judging Difficulty
DIPF | Leibniz Institute for Research and Information in Education(DIPF | 莱布尼茨教育与研究信息研究所);Centre for International Student Assessment (ZIB)(国际学生评估中心(ZIB));Faculty of Computer Science, Goethe University Frankfurt(法兰克福大学计算机科学学院);Chemnitz University of Technology(开姆尼茨工业大学)
Measuring Behavioural Signatures of Large Language Models through Psychometric Profiling
The Chinese University of Hong Kong, Shenzhen(香港中文大学(深圳));Central China Normal University(华中师范大学);Shandong Science and Technology Press, Shandong Publishing Group(山东科学技术出版社,山东出版集团);Bielefeld University(比勒费尔德大学);South China Normal University(华南师范大学);Frankfurt Institute for Advanced Studies(法兰克福高等研究院)
ValueBench: Towards Comprehensively Evaluating Value Orientations and Understanding of Large Language Models
Peking University(北京大学);National Key Laboratory of General Artificial Intelligence(通用人工智能国家重点实验室);School of Intelligence Science and Technology, Peking University(北京大学智能科学与技术学院);Department of Sociology, Peking University(北京大学社会学系);School of Psychological and Cognitive Sciences, Peking University(北京大学心理与认知科学学院);PKU-Wuhan Institute for Artificial Intelligence(北京大学武汉人工智能研究院)
PATCH! {P}sychometrics-{A}ssis{T}ed Ben{CH}marking of Large Language Models against Human Populations: A Case Study of Proficiency in 8th Grade Mathematics
Generative Personality Simulation via Theory-Informed Structured Interview
Department of Psychological Sciences & 2 Department of Computer Science, Rice University(1 心理学系 & 2 计算机科学系,莱斯大学);Department of Electrical and Computer Engineering, Northeastern University(3 电气与计算机工程系,东北大学);Department of Computer Science, Johns Hopkins University(4 计算机科学系,约翰霍普金斯大学);Microsoft Research Asia(微软亚洲研究院)
Position: AI Evaluation Should Learn from How We Test Humans
State Key Laboratory of Cognitive Intelligence, University of Science and Technology of China(中国科学技术大学认知智能国家重点实验室);University of California, Berkeley(加州大学伯克利分校);iFLYTEK Co., Ltd(科大讯飞股份有限公司)
Discovering Semantic Structures in Psychological Scales: A Response-Free Front-End to Efficient Simplification
Department of Psychological and Cognitive Sciences, Tsinghua University, Beijing, China(心理学与认知科学系,清华大学,北京,中国);Shenzhen International Graduate School, Tsinghua University, Shenzehen, China(深圳国际研究生院,清华大学,深圳,中国);Faculty of Psychology, Beijing Normal University, Beijing, China(心理学系,北京师范大学,北京,中国);Beijing Key Laboratory of Applied Experimental Psychology, National Demonstration Center for Experimental Psychology Education, Faculty of Psychology, Beijing Normal University, Beijing, China(应用实验心理学北京市重点实验室,国家级实验心理学教育示范中心,心理学系,北京师范大学,北京,中国)
BRIDGE: Predicting Human Task Completion Time From Model Performance
Mila - Quebec AI Institute(魁北克人工智能研究所);McGill University(麦吉尔大学);Polytechnique Montréal(蒙特利尔理工学院);Periodic Labs(Periodic实验室);Canada CIFAR AI Chair(加拿大CIFAR人工智能主席);ServiceNow Research(ServiceNow研究)